[release-4.16] OCPBUGS-52952: Add ipsec connect wait service - #4931
Conversation
When node goes for a reboot on an IPsec enabled cluster, once it comes up, after ovn-ipsec-host pod is deployed, the ovs-monitor-ipsec process deployed by the pod parses /etc/ipsec.d/openshift.conf file, makes pluto daemon to establish IKE SAs with peer nodes, but these are established after kubelet is started, workload pods scheduled on this node would fail communicating with other node pods until IPsec SAs are established. So this commit adds wait-for-ipsec-connect.service systemd service which depends on ipsecenabler.service created by IPsec machine config. This new service loads existing IPsec connections created by OVN/OVS into pluto daemon with "auto=start" option and waits upto 60s until IPsec tunnels are established. This gives a better chance IPsec SAs are established even before kubelet is started and when ovn-ipsec-host pod comes up later, it doesn't have to do anything for existing IPsec connections. We derived total wait time 60s based on the testing from 6 and 32 nodes cluster. In 6 node cluster, it took mostly about 2s (in worst case 4s), In 32 node cluster, it took mostly about 2s or 4s (in worst case 14s) to get IPsec connections up with peer nodes. The wait-for-ipsec-connect.service service is added into the base template to avoid two reboots during upgrade if it goes into IPsec machine configs rendered by the network operator. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 2417566) (cherry picked from commit 16e81a4) (cherry picked from commit 77e4ef2)
We noticed pluto is tearing down established IPsec connections in parallel with crio stopping all pod containers which includes stopping api server pod container. It happens when node reboot initiated for rendering new machine configs at the time of OCP upgrade. This creates api connection disruptions in the cluster, these disruptions are generating events, caught by origin monitor tests and failing IPsec upgrade CI lane and it may also cause noticeable temporary pod traffic failure during upgrade for IPsec enabled cluster. Hence this commit adds Before=crio.service dependency on the ipsec.service so that pluto daemon is stopped after the shutdown of crio service, all pod containers are stopped on the node. This gives enough room for clients to gracefully move to another control plane node for API connections. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 1fa5eaa) (cherry picked from commit 680d1ce) (cherry picked from commit 2e46d76)
openshift-ci-robot
added
jira/severity-moderate
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-52952, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/invalid-bug
|
/assign @huiran0826 @anuragthehatter |
|
/label qe-approved |
openshift-ci
Bot
added
the
qe-approved
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-52952, which is valid. 7 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/assign @djoshy |
|
/label backport-risk-assessed /lgtm |
openshift-ci
Bot
added
the
backport-risk-assessed
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: djoshy, pperiyasamy The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/label cherry-pick-approved |
|
@huiran0826: Can not set label cherry-pick-approved: Must be member in one of these teams: [] DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/label cherry-pick-approved |
openshift-ci
Bot
added
the
cherry-pick-approved
|
@pperiyasamy: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@pperiyasamy: Jira Issue OCPBUGS-52952: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-52952 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[ART PR BUILD NOTIFIER] Distgit: ose-machine-config-operator |
|
Fix included in accepted release 4.16.0-0.nightly-2025-04-02-211503 |
14 participants
Manual cherry-pick of #4930, No conflicts.