[release-4.17] OCPBUGS-52951: Unexpected Behavior During Cluster Upgrade for the ovn-ipsec-host pods - #2654
Conversation
openshift-ci-robot
added
jira/valid-reference
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-50582, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
pperiyasamy
changed the title
openshift-ci-robot
removed
the
jira/valid-reference
|
@pperiyasamy: No Jira issue is referenced in the title of this pull request. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
@huiran0826 This PR may help for your upgrade testing for #2644. |
pperiyasamy
changed the title
openshift-ci-robot
added
jira/severity-moderate
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-52951, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
09801dc to
6187ec8
Compare
|
/assign @anuragthehatter |
|
Looks good on QE tests as per steps |
|
/label qe-approved |
openshift-ci
Bot
added
the
qe-approved
|
@pperiyasamy: This pull request references Jira Issue OCPBUGS-52951, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@huiran0826: This pull request references Jira Issue OCPBUGS-52951, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
This commit captures machine config delete event, watches machine config pool while removing machine config from the cluster and reflect machine config pool status conditions into network status. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit b3c03e4) (cherry picked from commit 717548e)
When network operator owned machine config is deleted, though mco is still deleting the rendered machine config from the machine config pool, there is no way to lookup owner reference on the machine config object upon network operator pod reboot caused by node reboot. This commit writes machine config rendering status onto network.operator.openshift.io/last-seen-machineconfig-state network co annotation and this is being looked up to report status accordingly. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 080f445) (cherry picked from commit 79e7d3e)
This commit removes unnecessary initialization of machine config cache while handling events, cleans up cache entry at the right time, better handling of variable names, errors and adds few doc comments. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit b99b606) (cherry picked from commit 541ca8f)
When a machine config pool is in a paused state, then it doesn't process any machine config. so during legacy IPsec upgrade (4.14->4.15), IPsec machine configs may not be installed on the nodes when its pool is in paused state. In those cases the network operator continues to render older IPsec daemonsets which blocks network components from not getting upgraded to newer versions. Hence this commit renders newer IPsec daemonsets immediately, with new IPsecCheckForLibreswan check ensures one of the pods serves IPsec for the node. When MCPs are fully rolled out with ipsec machine config, then it goes ahead with rendering only host flavored IPsec daemonset. It brings in new behavior on IPsec daemonset rendering during IPsec deployment, upgrade and node reboot scenarios. 1. Users would notice both daemonsets being rendered at the time of IPsec install (or) upgrade for a temporary period until IPsec machine configs are fully deployed. 2. At the time of node reboot or machine config pool goes into progressing state, both demonsets being rendered. In this scenario, the containerized ipsec daemonset pods are dormant. 3. It removes legacy upgrade case as every upgrade would be considered as the same with this approach. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 37693e9) (cherry picked from commit 2ddaf1c)
The previous commit c12cdd4 renders ipsec host daemonset even before machine config are deployed on the node, the ipsec host paths /usr/sbin/ipsec and /usr/libexec/ipsec on the are not available until libreswan installed on the node. So pod fails to come up and goes into pending state because host volume doesn't exist and network co is blocked from moving from a progressing state to an available state. To fix this problem, mount its top level directory into ovn-ipsec container which is a system level directory which is always present. During OCP upgrade from previous 4.15.z to this fix release, with worker pool is in paused state, both network and machine config cluster operator are upgraded into this fix release, the new host ipsec deployment is rendered which has libreswan 4.6 package installed on the container. Since worker node are paused, the host is still having libreswan 4.9 package installed and pluto is with this version. But this is not a problem with this commit, we mount /usr/sbin and /usr/libexec directories, The /usr/sbin/ipsec, /usr/libexec/ipsec/addconn and /usr/libexec/ipsec/_stackmanager commands are used inside the container. The ipsec and _stackmanager are bash scripts which should work without a problem. The addconn is a "C" compiled binary having some dynamic library dependencies and the container uses this command to validate /etc/ipsec.conf file. This must also work because /usr/libexec/ipsec mount was there previously as well. sh-5.1# ldd /usr/sbin/ipsec not a dynamic executable sh-5.1# ldd /usr/libexec/ipsec/_stackmanager not a dynamic executable sh-5.1# ldd /usr/libexec/ipsec/addconn linux-vdso.so.1 (0x00007ffc87bf7000) libunbound.so.8 => /lib64/libunbound.so.8 (0x00007f809f5f3000) libldns.so.3 => /lib64/libldns.so.3 (0x00007f809f58b000) libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007f809f56b000) libc.so.6 => /lib64/libc.so.6 (0x00007f809f200000) libssl.so.3 => /lib64/libssl.so.3 (0x00007f809f4c5000) libprotobuf-c.so.1 => /lib64/libprotobuf-c.so.1 (0x00007f809f4ba000) libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007f809f45f000) libpython3.9.so.1.0 => /lib64/libpython3.9.so.1.0 (0x00007f809ee00000) libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f809e800000) libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f809f435000) /lib64/ld-linux-x86-64.so.2 (0x00007f809f7b3000) libm.so.6 => /lib64/libm.so.6 (0x00007f809ed25000) libz.so.1 => /lib64/libz.so.1 (0x00007f809f41b000) Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 865ac69) (cherry picked from commit 91a6b78)
The CNO started using machine configs from 4.15 for IPsec deployment, so adding a check for machine config operator to be at least >= 4.15 to roll out IPsec machine configs. Otherwise during OCP 4.14->4.15 upgrade, even before MCO is upgraded to 4.15, IPsec machine configs are rolled out, it uses ipsec extension from 4.14 version to install packages, installs libreswan 4.9 version on the node intermeditately. So this MCO version check ensures IPsec machine configs are rendered after MCO is upgraded to 4.15 and nodes get desired libreswan version 4.6. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit cb99bc3) (cherry picked from commit c8f4cf8)
The rpm db directory is different on rhcos and rhel workers, so mounting /usr/share/rpm directory will not work for rhel worker nodes. To avoid this, this commit checks on ipsec systemd service on the host to decide which ipsec deployment to be active or dormant. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 11c08e6) (cherry picked from commit 1abd78b)
6187ec8 to
9c969cc
Compare
|
/lgtm |
openshift-ci
Bot
added
the
backport-risk-assessed
openshift-ci
Bot
assigned
asood-rh,
eurijon,
jechen0648,
meinali566,
mffiedler,
qiowang721 and
rbbratta
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jcaamano, pperiyasamy The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/label cherry-pick-approved |
openshift-ci
Bot
added
the
cherry-pick-approved
|
@pperiyasamy: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@pperiyasamy: Jira Issue OCPBUGS-52951: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-52951 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[ART PR BUILD NOTIFIER] Distgit: cluster-network-operator |
|
Fix included in accepted release 4.18.0-0.nightly-2025-04-17-181105 |
14 participants
Backport of #2644.
Resolved Conflicts:
5762c31:
pkg/controller/statusmanager/status_manager.go(needed to importk8s.io/apimachinery/pkg/util/sets)5762c31 and bf7e75d:
pkg/network/ovn_kubernetes_test.go(conflict with existing import packages)./assign @huiran0826