[release-4.18] OCPBUGS-52949, OCPBUGS-50582: Unexpected Behavior During Cluster Upgrade for the ovn-ipsec-host pods by pperiyasamy · Pull Request #2644 · openshift/cluster-network-operator

pperiyasamy · 2025-02-12T08:55:34Z

Manually cherry picked master PRs #2611, #2621, #2628, #2383 and #2454. These commits containing necessary fixes to solve IPsec upgrade issue when a machine config pool in paused state and temporary pod traffic hit at the time of the upgrade.

No conflicts seen.

openshift-ci-robot · 2025-02-12T08:55:40Z

@pperiyasamy: This pull request references Jira Issue OCPBUGS-50582, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-50582 to depend on a bug targeting a version in 4.19.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This PR does the backport of following fixes to prevent unnecessary ipsec service restart, ip xfrm state policy cleanups while bringing up ipsec-host pod. This would potentially avoid reestablishment of IKE SAs during ipsec pod restarts and let OVN networking pods traffic go on without any packet drops.

There is an incorrect check in ipsec pod clean up logic which removes /etc/ipsec.d/openshift.conf file, ip xfrm state and policy entries in all cases, but these must be removed only when ipsec mode is changed from full to external or disabled.

We don't need narrowing=yes option to be set explicitly anymore because system default crypto policies are commented out now, otherwise TS_UNACCEPTABLE error is seen temporarily at the time of ipsec service restart.

The IPsec service restart is needed only at the time of specific IPsec config changes, so doing it only at the time commenting out default crypto-policies conf file.

After IPsec is deployed, when machine config pools goes into progressing/degraded state while processing some other machine machine configs (or) node reboot scenarios, CNO deletes IPsec pods which is known behavior as per changes in the commit 6aa2f0f, but it also accidentally disabling IPsec in OVN which is not an expected behavior. This causes ovs-monitor-ipsec to refresh existing ipsec connections unnecessarily when IPsec pod comes up (once machine config pool settles) as it is not able to find remote_name from the tunnel.

It also removes dead code related to IPsec 4.13 upgrade which is no longer valid for upgrade scenarios beyond >= 4.15. This change is intended to be backported until 4.15 so still keeping legacy upgrade (4.14->4.15) scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pperiyasamy · 2025-02-12T09:23:23Z

/jira refresh

openshift-ci-robot · 2025-02-12T09:23:31Z

@pperiyasamy: This pull request references Jira Issue OCPBUGS-50582, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.0) matches configured target version for branch (4.18.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-50616 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-50616 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pperiyasamy · 2025-02-12T09:24:11Z

/assign @jcaamano @huiran0826

openshift-ci-robot · 2025-02-12T09:54:45Z

@pperiyasamy: This pull request references Jira Issue OCPBUGS-50582, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.0) matches configured target version for branch (4.18.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-50616 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-50616 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @huiran0826

Details

In response to this:

This PR does the backport of following fixes to prevent unnecessary ipsec service restart, ip xfrm state policy cleanups while bringing up ipsec-host pod. This would potentially avoid reestablishment of IKE SAs during ipsec pod restarts and let OVN networking pods traffic go on without any packet drops.

There is an incorrect check in ipsec pod clean up logic which removes /etc/ipsec.d/openshift.conf file, ip xfrm state and policy entries in all cases, but these must be removed only when ipsec mode is changed from full to external or disabled.

We don't need narrowing=yes option to be set explicitly anymore because system default crypto policies are commented out now, otherwise TS_UNACCEPTABLE error is seen temporarily at the time of ipsec service restart.

The IPsec service restart is needed only at the time of specific IPsec config changes, so doing it only at the time commenting out default crypto-policies conf file.

After IPsec is deployed, when machine config pools goes into progressing/degraded state while processing some other machine machine configs (or) node reboot scenarios, CNO deletes IPsec pods which is known behavior as per changes in the commit 6aa2f0f, but it also accidentally disabling IPsec in OVN which is not an expected behavior. This causes ovs-monitor-ipsec to refresh existing ipsec connections unnecessarily when IPsec pod comes up (once machine config pool settles) as it is not able to find remote_name from the tunnel.

It also removes dead code related to IPsec 4.13 upgrade which is no longer valid for upgrade scenarios beyond >= 4.15. This change is intended to be backported until 4.15 so still keeping legacy upgrade (4.14->4.15) scenarios.

Manually cherry picked master commits 864bdc5, ece9fbb, ea1d489, 4e57dcd and ff0b147, no conflicts seen.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

huiran0826 · 2025-02-28T13:33:05Z

Following verification steps in https://issues.redhat.com/browse/OCPBUGS-50616 and pre-merge testing result looks good.
/label qe-approved

openshift-ci-robot · 2025-02-28T13:33:15Z

@pperiyasamy: This pull request references Jira Issue OCPBUGS-50582, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.z) matches configured target version for branch (4.18.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-50616 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-50616 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @huiran0826

Details

In response to this:

This PR does the backport of following fixes to prevent unnecessary ipsec service restart, ip xfrm state policy cleanups while bringing up ipsec-host pod. This would potentially avoid reestablishment of IKE SAs during ipsec pod restarts and let OVN networking pods traffic go on without any packet drops.

There is an incorrect check in ipsec pod clean up logic which removes /etc/ipsec.d/openshift.conf file, ip xfrm state and policy entries in all cases, but these must be removed only when ipsec mode is changed from full to external or disabled.

We don't need narrowing=yes option to be set explicitly anymore because system default crypto policies are commented out now, otherwise TS_UNACCEPTABLE error is seen temporarily at the time of ipsec service restart.

The IPsec service restart is needed only at the time of specific IPsec config changes, so doing it only at the time commenting out default crypto-policies conf file.

After IPsec is deployed, when machine config pools goes into progressing/degraded state while processing some other machine machine configs (or) node reboot scenarios, CNO deletes IPsec pods which is known behavior as per changes in the commit 6aa2f0f, but it also accidentally disabling IPsec in OVN which is not an expected behavior. This causes ovs-monitor-ipsec to refresh existing ipsec connections unnecessarily when IPsec pod comes up (once machine config pool settles) as it is not able to find remote_name from the tunnel.

It also removes dead code related to IPsec 4.13 upgrade which is no longer valid for upgrade scenarios beyond >= 4.15. This change is intended to be backported until 4.15 so still keeping legacy upgrade (4.14->4.15) scenarios.

Manually cherry picked master commits 864bdc5, ece9fbb, ea1d489, 4e57dcd and ff0b147, no conflicts seen.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pperiyasamy · 2025-03-03T11:56:08Z

/retest-required

openshift-ci-robot · 2025-03-11T09:15:09Z

@pperiyasamy: This pull request references Jira Issue OCPBUGS-52949, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected dependent Jira Issue OCPBUGS-52280 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-50582, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.z) matches configured target version for branch (4.18.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-50616 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-50616 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @huiran0826

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Manually cherry picked master PRs #2611, #2621, #2628, #2383 and #2454. These commits containing necessary fixes to solve IPsec upgrade issue when a machine config pool in paused state and temporary pod traffic hit at the time of the upgrade.

No conflicts seen.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

trozet · 2025-03-12T15:31:03Z

/hold

until 2454 merges

There is an incorrect check while cleaning up ipsec state upon deleting ipsec pod which removes states in all cases, so this fix removes state only when ipsec mode is not full mode. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 864bdc5)

jcaamano · 2025-03-26T10:34:26Z

/jira refresh

openshift-ci-robot · 2025-03-26T10:34:33Z

@jcaamano: This pull request references Jira Issue OCPBUGS-52949, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.z) matches configured target version for branch (4.18.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-52280 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-52280 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @anuragthehatter

This pull request references Jira Issue OCPBUGS-50582, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.z) matches configured target version for branch (4.18.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-50616 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-50616 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @huiran0826

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pperiyasamy · 2025-03-26T10:41:07Z

/label acknowledge-critical-fixes-only

this is a critical fix and must be backported into 4.15 ASAP for a customer bug https://issues.redhat.com/browse/OCPBUGS-36688,

jcaamano · 2025-03-26T10:42:48Z

/lgtm
/approve
/label backport-risk-assessed

openshift-ci · 2025-03-26T10:43:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcaamano, pperiyasamy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jcaamano]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

huiran0826 · 2025-03-26T11:04:13Z

/label cherry-pick-approved

pperiyasamy · 2025-03-26T11:13:04Z

/retest-required

openshift-ci-robot · 2025-03-26T13:36:53Z

/retest-required

Remaining retests: 0 against base HEAD 8cbda2d and 2 for PR HEAD 1abd78b in total

openshift-ci-robot · 2025-03-26T21:32:08Z

/retest-required

Remaining retests: 0 against base HEAD 8cbda2d and 2 for PR HEAD 1abd78b in total

pperiyasamy · 2025-03-27T06:32:59Z

/retest-required

pperiyasamy · 2025-03-27T07:50:19Z

/retest-required

openshift-ci-robot · 2025-03-27T09:33:30Z

/retest-required

Remaining retests: 0 against base HEAD 8cbda2d and 2 for PR HEAD 1abd78b in total

openshift-ci · 2025-03-27T13:17:17Z

@pperiyasamy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-hypershift-ovn-kubevirt	`1abd78b`	link	false	`/test e2e-aws-hypershift-ovn-kubevirt`
ci/prow/e2e-aws-ovn-ipsec-upgrade	`1abd78b`	link	false	`/test e2e-aws-ovn-ipsec-upgrade`
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6	`1abd78b`	link	false	`/test e2e-vsphere-ovn-dualstack-primaryv6`
ci/prow/security	`1abd78b`	link	false	`/test security`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-03-27T14:01:15Z

/retest-required

Remaining retests: 0 against base HEAD 8cbda2d and 2 for PR HEAD 1abd78b in total

openshift-ci-robot · 2025-03-27T16:57:55Z

@pperiyasamy: Jira Issue OCPBUGS-52949: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-52949 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-50582: Some pull requests linked via external trackers have merged:

openshift/cluster-network-operator#2644

The following pull requests linked via external trackers have not merged:

openshift/cluster-network-operator#2654 is open

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-50582 has not been moved to the MODIFIED state.

Details

In response to this:

Manually cherry picked master PRs #2611, #2621, #2628, #2383 and #2454. These commits containing necessary fixes to solve IPsec upgrade issue when a machine config pool in paused state and temporary pod traffic hit at the time of the upgrade.

No conflicts seen.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2025-03-27T21:20:59Z

[ART PR BUILD NOTIFIER]

Distgit: cluster-network-operator
This PR has been included in build cluster-network-operator-container-v4.18.0-202503272032.p0.g670c2e3.assembly.stream.el9.
All builds following this will include this PR.

openshift-merge-robot · 2025-04-30T14:19:14Z

Fix included in accepted release 4.18.0-0.nightly-2025-04-17-181105

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Feb 12, 2025

openshift-ci Bot requested review from abhat and kyrtapz February 12, 2025 08:57

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Feb 12, 2025

openshift-ci Bot requested a review from anuragthehatter February 12, 2025 09:23

openshift-ci Bot assigned huiran0826 and jcaamano Feb 12, 2025

openshift-ci Bot requested a review from huiran0826 February 12, 2025 09:54

pperiyasamy mentioned this pull request Feb 26, 2025

[release-4.17] OCPBUGS-52951: Unexpected Behavior During Cluster Upgrade for the ovn-ipsec-host pods #2654

Merged

openshift-ci Bot added the qe-approved Signifies that QE has signed off on this PR label Feb 28, 2025

pperiyasamy force-pushed the ipsec-graceful-cleanup-4.18 branch from 813104e to f6a59e0 Compare March 11, 2025 08:10

pperiyasamy changed the title ~~[release-4.18] OCPBUGS-50582: Graceful cleanup of IPsec states~~ [release-4.18] OCPBUGS-52949, OCPBUGS-50582: Graceful cleanup of IPsec states Mar 11, 2025

pperiyasamy changed the title ~~[release-4.18] OCPBUGS-52949, OCPBUGS-50582: Graceful cleanup of IPsec states~~ [release-4.18] OCPBUGS-52949, OCPBUGS-50582: Unexpected Behavior During Cluster Upgrade for the ovn-ipsec-host pods Mar 11, 2025

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 12, 2025

openshift-ci Bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Mar 26, 2025

openshift-ci Bot assigned anuragthehatter, asood-rh, eurijon, jechen0648, meinali566, mffiedler, qiowang721 and rbbratta Mar 26, 2025

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 26, 2025

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 26, 2025

openshift-ci Bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 26, 2025

openshift-merge-bot Bot merged commit 670c2e3 into openshift:release-4.18 Mar 27, 2025

Uh oh!

Conversation

pperiyasamy commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Feb 12, 2025

Uh oh!

pperiyasamy commented Feb 12, 2025

Uh oh!

openshift-ci-robot commented Feb 12, 2025

Uh oh!

pperiyasamy commented Feb 12, 2025

Uh oh!

openshift-ci-robot commented Feb 12, 2025

Uh oh!

huiran0826 commented Feb 28, 2025

Uh oh!

openshift-ci-robot commented Feb 28, 2025

Uh oh!

pperiyasamy commented Mar 3, 2025

Uh oh!

openshift-ci-robot commented Mar 11, 2025

Uh oh!

trozet commented Mar 12, 2025

Uh oh!

jcaamano commented Mar 26, 2025

Uh oh!

openshift-ci-robot commented Mar 26, 2025

Uh oh!

pperiyasamy commented Mar 26, 2025

Uh oh!

jcaamano commented Mar 26, 2025

Uh oh!

openshift-ci Bot commented Mar 26, 2025

Uh oh!

huiran0826 commented Mar 26, 2025

Uh oh!

pperiyasamy commented Mar 26, 2025

Uh oh!

openshift-ci-robot commented Mar 26, 2025

Uh oh!

openshift-ci-robot commented Mar 26, 2025

Uh oh!

pperiyasamy commented Mar 27, 2025

Uh oh!

pperiyasamy commented Mar 27, 2025

Uh oh!

openshift-ci-robot commented Mar 27, 2025

Uh oh!

openshift-ci Bot commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 27, 2025

Uh oh!

openshift-ci-robot commented Mar 27, 2025

Uh oh!

openshift-bot commented Mar 27, 2025

Uh oh!

openshift-merge-robot commented Apr 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

pperiyasamy commented Feb 12, 2025 •

edited

Loading

openshift-ci Bot commented Mar 27, 2025 •

edited

Loading