[release-4.15] OCPBUGS-36688: Add ipsec connect wait service - #4937
Conversation
When node goes for a reboot on an IPsec enabled cluster, once it comes up, after ovn-ipsec-host pod is deployed, the ovs-monitor-ipsec process deployed by the pod parses /etc/ipsec.d/openshift.conf file, makes pluto daemon to establish IKE SAs with peer nodes, but these are established after kubelet is started, workload pods scheduled on this node would fail communicating with other node pods until IPsec SAs are established. So this commit adds wait-for-ipsec-connect.service systemd service which depends on ipsecenabler.service created by IPsec machine config. This new service loads existing IPsec connections created by OVN/OVS into pluto daemon with "auto=start" option and waits upto 60s until IPsec tunnels are established. This gives a better chance IPsec SAs are established even before kubelet is started and when ovn-ipsec-host pod comes up later, it doesn't have to do anything for existing IPsec connections. We derived total wait time 60s based on the testing from 6 and 32 nodes cluster. In 6 node cluster, it took mostly about 2s (in worst case 4s), In 32 node cluster, it took mostly about 2s or 4s (in worst case 14s) to get IPsec connections up with peer nodes. The wait-for-ipsec-connect.service service is added into the base template to avoid two reboots during upgrade if it goes into IPsec machine configs rendered by the network operator. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 2417566) (cherry picked from commit 16e81a4) (cherry picked from commit 77e4ef2) (cherry picked from commit c31fd9c)
We noticed pluto is tearing down established IPsec connections in parallel with crio stopping all pod containers which includes stopping api server pod container. It happens when node reboot initiated for rendering new machine configs at the time of OCP upgrade. This creates api connection disruptions in the cluster, these disruptions are generating events, caught by origin monitor tests and failing IPsec upgrade CI lane and it may also cause noticeable temporary pod traffic failure during upgrade for IPsec enabled cluster. Hence this commit adds Before=crio.service dependency on the ipsec.service so that pluto daemon is stopped after the shutdown of crio service, all pod containers are stopped on the node. This gives enough room for clients to gracefully move to another control plane node for API connections. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 1fa5eaa) (cherry picked from commit 680d1ce) (cherry picked from commit 2e46d76) (cherry picked from commit b09c4ac)
openshift-ci-robot
added
jira/severity-moderate
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-36688, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/assign @huiran0826 @anuragthehatter |
|
@pperiyasamy: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Looks good on QE tests /label qe-approved |
openshift-ci
Bot
added
the
qe-approved
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-36688, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-36688, which is valid. 7 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/assign @djoshy |
|
/lgtm /label backport-risk-assessed |
openshift-ci
Bot
added
the
backport-risk-assessed
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: djoshy, pperiyasamy The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/label cherry-pick-approved |
openshift-ci
Bot
added
the
cherry-pick-approved
|
@pperiyasamy: Jira Issue OCPBUGS-36688: Some pull requests linked via external trackers have merged: The following pull requests linked via external trackers have not merged: These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-36688 has not been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[ART PR BUILD NOTIFIER] Distgit: ose-machine-config-operator |
[1] contains [2,3], picking up [4,5]. [1]: https://amd64.ocp.releases.ci.openshift.org/releasestream/4-stable/release/4.15.49 [2]: https://issues.redhat.com/browse/OCPBUGS-36688 [3]: https://issues.redhat.com/browse/OCPBUGS-43099 [4]: openshift/machine-config-operator#4937 [5]: openshift/cluster-network-operator#2658
13 participants
Manual cherry-pick of 4.16 PR #4931. No conflicts.