Skip to content

SDN-5330: Wait enough for IPsec tunnels to establish before kubelet start - #2606

Closed
pperiyasamy wants to merge 2 commits into
openshift:masterfrom
pperiyasamy:ipsec-post-start
Closed

SDN-5330: Wait enough for IPsec tunnels to establish before kubelet start#2606
pperiyasamy wants to merge 2 commits into
openshift:masterfrom
pperiyasamy:ipsec-post-start

Conversation

@pperiyasamy

Copy link
Copy Markdown
Member

When node goes for a reboot on an IPsec enabled cluster, once it comes up, libreswan parses /etc/ipsec.d/openshift.conf file and establishes SAs with peers and it may be still in progress even after kubelet is started, pod scheduled on this node would fail communicating with other pods until IPsec tunnels are established. So this commit makes ipsecenabler systemd service to wait upto 2 minutes until IPsec tunnels are established and this would give better chance for ipsec CI lanes to run fine without disruptive events.

The butane configs are translated into machine configs with command butane <file>.bu -o <file>.yaml and only storage portion is copied manually into IPsec machine config files.

@openshift-ci
openshift-ci Bot requested review from abhat and tssurya December 18, 2024 12:42
@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade
/test e2e-aws-ovn-ipsec-serial

@pperiyasamy

Copy link
Copy Markdown
Member Author

/assign @trozet @jcaamano

@pperiyasamy

Copy link
Copy Markdown
Member Author

somehow ipsec-connect-wait.sh systemd poststart script always sees /etc/ipsec.d/openshift.conf file as unavailable while running, need to troubleshoot why that's happening.

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade
/test e2e-aws-ovn-ipsec-serial

@openshift-ci

openshift-ci Bot commented Dec 19, 2024

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pperiyasamy
Once this PR has been reviewed and has the lgtm label, please ask for approval from jcaamano. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade
/test e2e-aws-ovn-ipsec-serial

1 similar comment
@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade
/test e2e-aws-ovn-ipsec-serial

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade
/test e2e-aws-ovn-ipsec-serial

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy pperiyasamy changed the title Wait enough for IPsec tunnels to establish before kubelet start SDN-5330: Wait enough for IPsec tunnels to establish before kubelet start Jan 8, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 8, 2025
@openshift-ci-robot

openshift-ci-robot commented Jan 8, 2025

Copy link
Copy Markdown
Contributor

@pperiyasamy: This pull request references SDN-5330 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

Details

In response to this:

When node goes for a reboot on an IPsec enabled cluster, once it comes up, libreswan parses /etc/ipsec.d/openshift.conf file and establishes SAs with peers and it may be still in progress even after kubelet is started, pod scheduled on this node would fail communicating with other pods until IPsec tunnels are established. So this commit makes ipsecenabler systemd service to wait upto 2 minutes until IPsec tunnels are established and this would give better chance for ipsec CI lanes to run fine without disruptive events.

The butane configs are translated into machine configs with command butane <file>.bu -o <file>.yaml and only storage portion is copied manually into IPsec machine config files.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade
/test e2e-aws-ovn-ipsec-serial
/testwith openshift/cluster-network-operator/master/e2e-ovn-ipsec-step-registry openshift/origin#29232

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-serial

1 similar comment
@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-serial

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-ovn-ipsec-step-registry openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

The ipsec-connect-wait.sh script with auto=start parameter brings up ipsec connections timely before kubelet, but still monitor test failures are seen, so trying out tweaking ipsec tests to find out the root cause.

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

1 similar comment
@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

2 similar comments
@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-ovn-ipsec-step-registry openshift/origin#29232

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-ovn-ipsec-step-registry openshift/origin#29232

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-ovn-ipsec-step-registry openshift/origin#29232

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

When node goes for a reboot on an IPsec enabled cluster, once it comes up, libreswan
parses /etc/ipsec.d/openshift.conf file and establishes SAs with peers and it may be
still in progress even after kubelet is started, pod scheduled on this node would fail
communicating with other pods until IPsec tunnels are established.
So this commit makes ipsecenabler systemd service to wait upto 2 minutes until IPsec
tunnels are established and this would give better chance for ipsec CI lanes to run
fine without disruptive events.

The butane configs are translated into machine configs with command `butane <file>.bu -o <file>.yaml`
and only storage portion is copied manually into IPsec machine config files.

Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>
Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>
@pperiyasamy

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn-ipsec-upgrade

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-aws-ovn-ipsec-serial openshift/origin#29437

@openshift-ci

openshift-ci Bot commented Feb 12, 2025

Copy link
Copy Markdown
Contributor

@pperiyasamy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-ipsec-serial 8d0db5b link false /test e2e-aws-ovn-ipsec-serial
ci/prow/4.18-upgrade-from-stable-4.17-e2e-aws-ovn-upgrade b1aec41 link false /test 4.18-upgrade-from-stable-4.17-e2e-aws-ovn-upgrade
ci/prow/4.18-upgrade-from-stable-4.17-e2e-azure-ovn-upgrade b1aec41 link false /test 4.18-upgrade-from-stable-4.17-e2e-azure-ovn-upgrade
ci/prow/e2e-aws-hypershift-ovn-kubevirt 7d35250 link false /test e2e-aws-hypershift-ovn-kubevirt
ci/prow/e2e-aws-ovn-ipsec-upgrade 7d35250 link false /test e2e-aws-ovn-ipsec-upgrade
ci/prow/e2e-azure-ovn 7d35250 link false /test e2e-azure-ovn
ci/prow/security 7d35250 link false /test security
ci/prow/e2e-aws-ovn-hypershift-conformance 7d35250 link true /test e2e-aws-ovn-hypershift-conformance
ci/prow/4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade 7d35250 link false /test 4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6 7d35250 link false /test e2e-vsphere-ovn-dualstack-primaryv6

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@pperiyasamy

Copy link
Copy Markdown
Member Author

/testwith openshift/cluster-network-operator/master/e2e-ovn-ipsec-step-registry openshift/origin#29232

@pperiyasamy

Copy link
Copy Markdown
Member Author

Closing this in favor of openshift/machine-config-operator#4854. The MCO PR optimizes things with single node reboot during ipsec upgrade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants