[release-4.18] CORENET-5568: Stabilize IPsec E2E tests - #29632
[release-4.18] CORENET-5568: Stabilize IPsec E2E tests #29632pperiyasamy wants to merge 6 commits into
Conversation
The ipsec test suite takes about 5-6 hours to finish running the tests for e2e-aws-ovn-ipsec-serial CI lane, Hence this commit optimizes ipsec tests so that it doesn't cause much of node reboots when changing ipsec modes. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 1ee6b62)
Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 3d52d74)
This attempts to fix following error seen from ipsec test run:
I1118 11:08:34.394429 461 delete.go:70] Wait up to 5m0s for pod "ipsec-tcpdump-hostpod-vftlk" to be fully deleted
I1118 11:09:07.535653 461 exec_util.go:59] ExecWithOptions {Command:[/bin/sh -c cat /release-manifests/image-references] Namespace:e2e-test-ipsec-48sdq PodName:extract-release-imagerefs ContainerName:imagerefs Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
I1118 11:09:07.536406 461 exec_util.go:66] ExecWithOptions: Clientset creation
I1118 11:09:07.536456 461 exec_util.go:83] ExecWithOptions: execute(POST https://api.ci-op-6ic49xd0-f3b46.aws-2.ci.openshift.org:6443/api/v1/namespaces/e2e-test-ipsec-48sdq/pods/extract-release-imagerefs/exec?command=%2Fbin%2Fsh&command=-c&command=cat+%2Frelease-manifests%2Fimage-references&container=imagerefs&container=imagerefs&stderr=true&stdout=true)
I1118 11:09:07.642284 461 exec_util.go:111] Exec stderr: ""
I1118 11:09:07.642418 461 exec_util.go:112] Unexpected error: failed to execute command in pod extract-release-imagerefs, container imagerefs: error sending request: Post "https://api.ci-op-6ic49xd0-f3b46.aws-2.ci.openshift.org:6443/api/v1/namespaces/e2e-test-ipsec-48sdq/pods/extract-release-imagerefs/exec?command=%2Fbin%2Fsh&command=-c&command=cat+%2Frelease-manifests%2Fimage-references&container=imagerefs&container=imagerefs&stderr=true&stdout=true": read tcp 10.131.68.7:51092->54.244.16.237:6443: read: connection reset by peer:
<*errors.errorString | 0xc006aabf10>:
error sending request: Post "https://api.ci-op-6ic49xd0-f3b46.aws-2.ci.openshift.org:6443/api/v1/namespaces/e2e-test-ipsec-48sdq/pods/extract-release-imagerefs/exec?command=%2Fbin%2Fsh&command=-c&command=cat+%2Frelease-manifests%2Fimage-references&container=imagerefs&container=imagerefs&stderr=true&stdout=true": read tcp 10.131.68.7:51092->54.244.16.237:6443: read: connection reset by peer
{
s: "error sending request: Post \"https://api.ci-op-6ic49xd0-f3b46.aws-2.ci.openshift.org:6443/api/v1/namespaces/e2e-test-ipsec-48sdq/pods/extract-release-imagerefs/exec?command=%2Fbin%2Fsh&command=-c&command=cat+%2Frelease-manifests%2Fimage-references&container=imagerefs&container=imagerefs&stderr=true&stdout=true\": read tcp 10.131.68.7:51092->54.244.16.237:6443: read: connection reset by peer",
}
[FAILED] in [It] - k8s.io/kubernetes@v1.31.1/test/e2e/framework/pod/exec_util.go:112 @ 11/18/24 11:09:07.701
Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>
(cherry picked from commit 7386625)
When IPsec tests configuring certificates into libreswan nss db for north south traffic via a machine config, it's rebooting worker nodes by default which still makes one of the following monitor test to fail. 4 unexpected clusteroperator state transitions during e2e test run. These did not match any known exceptions, so they cause this test-case to fail: Jan 15 19:44:18.575 E clusteroperator/kube-storage-version-migrator condition/Available reason/KubeStorageVersionMigrator_Deploying status/False KubeStorageVersionMigratorAvailable: Waiting for Deployment Jan 15 19:44:18.575 - 5s E clusteroperator/kube-storage-version-migrator condition/Available reason/KubeStorageVersionMigrator_Deploying status/False KubeStorageVersionMigratorAvailable: Waiting for Deployment Jan 15 20:06:29.820 E clusteroperator/kube-storage-version-migrator condition/Available reason/KubeStorageVersionMigrator_Deploying status/False KubeStorageVersionMigratorAvailable: Waiting for Deployment Jan 15 20:06:29.820 - 1s E clusteroperator/kube-storage-version-migrator condition/Available reason/KubeStorageVersionMigrator_Deploying status/False KubeStorageVersionMigratorAvailable: Waiting for Deployment 2 unwelcome but acceptable clusteroperator state transitions during e2e test run. These should not happen, but because they are tied to exceptions, the fact that they did happen is not sufficient to cause this test-case to fail: Jan 15 19:44:24.518 W clusteroperator/kube-storage-version-migrator condition/Available reason/AsExpected status/True All is well (exception: Available=True is the happy case) Jan 15 20:06:31.725 W clusteroperator/kube-storage-version-migrator condition/Available reason/AsExpected status/True All is well (exception: Available=True is the happy case) Actually it is not required to reboot the nodes just for configuring certs on the nss db. Hence adding nod edisruption machine configuration policy so that nodes are not rebooted while deploying certificates on the worker nodes. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit 14e7f6c)
When IPsec mode are changed across tests within IPsec test suite, it causes reboot of ovnkube-node daemonset pods, It's expected workload traffic would fail temporarily until pods are settle down after IPsec is properly configured in every node's OVN and OvS across the cluster. So we should not test ipsec mode change in the ipsec test suite and instead for every ipsec mode, there should be one CI lane, then in the test corresponding configuration and traffic must be tested. So this commit removes everything related to IPsec mode changes and having a single test which can be run from Full and External IPsec mode CI lanes. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit c53288d)
This commit fixes following test failures seen in CI runs. 1. [sig-auth] all workloads in ns/openshift-nmstate must set the 'openshift.io/required-scc' annotation https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29437/pull-ci-openshift-origin-master-e2e-aws-ovn-ipsec-serial/1879584910717489152 2. Wait enough to get nmstate-handler daemonset running state. [sig-network][Feature:IPsec] when using openshift ovn-kubernetes [BeforeAll] check traffic [apigroup:config.openshift.io] [Suite:openshift/network/ipsec] with IPsec in full mode [BeforeAll] github.com/openshift/origin/test/extended/networking/ipsec.go:449 [It] github.com/openshift/origin/test/extended/networking/ipsec.go:628 [FAILED] Unexpected error: <*errors.errorString | 0xc0015847e0>: failed to get nmstate handler running: context deadline exceeded { s: "failed to get nmstate handler running: context deadline exceeded", } occurred In [BeforeAll] at: github.com/openshift/origin/test/extended/networking/ipsec.go:464 @ 01/16/25 11:12:58.598 https://prow.ci.openshift.org/view/gs/test-platform-results/logs/multi-pr-openshift-cluster-network-operator-2606-openshift-origin-29437-e2e-aws-ovn-ipsec-serial/1879821657376296960 Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com> (cherry picked from commit e2e54ef)
|
/test e2e-aws-ovn-ipsec-serial |
|
/assign @martinkennelly @zshi-redhat |
|
/test e2e-aws-ovn-ipsec-serial |
|
/testwith openshift/origin/release-4.18/e2e-aws-ovn-ipsec-serial openshift/machine-config-operator#4967 |
martinkennelly
left a comment
martinkennelly
left a comment
There was a problem hiding this comment.
Lgtm .. Waiting on ci. Any changes you needed to make ? Ive only a cosmetic question.
| } | ||
|
|
||
| func findIPsecPodonNode(oc *exutil.CLI, nodeName string) (string, error) { | ||
| out, err := runOcWithRetry(oc.AsAdmin(), "get", |
There was a problem hiding this comment.
Kube client not available ?
There was a problem hiding this comment.
@martinkennelly of course kube client is available and it can be simplified with that, but i think we can do it with another PR. let us keep this backport as it is now. are you ok ?
|
@pperiyasamy: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/test e2e-aws-ovn-ipsec-serial |
|
/lgtm looked at the ipsec lane and saw it executed one ipsec test for 6min. |
|
/assign @dgoodwin |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgoodwin, martinkennelly, pperiyasamy The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/label backport-risk-assessed |
openshift-ci
Bot
added
the
backport-risk-assessed
pperiyasamy
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@pperiyasamy: This pull request references CORENET-5568 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.z" version, but no target version was set. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
Bot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
openshift-ci
Bot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
|
@openshift-bot: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
13 participants
Backport of PRs #28797, #29437 and #29452. No Conflicts.