AUTOSCALE-166: Add weekly karpenter core regression e2e periodic for aws-karpenter-p… - #82258
AUTOSCALE-166: Add weekly karpenter core regression e2e periodic for aws-karpenter-p…#82258fishereskew wants to merge 1 commit into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change adds generated periodic CI configuration for ChangesKarpenter periodic CI
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/aws-karpenter-provider-aws/openshift-aws-karpenter-provider-aws-main__periodics.yaml`:
- Around line 64-109: Run make update after adding the
e2e-aws-karpenter-core-regression periodic configuration, then include all
generated Prow job and metadata changes in the commit. Ensure the generated
artifacts reflect the new periodic job so it is scheduled.
- Around line 101-102: Update the upstream-e2e.sh invocation in the Karpenter
regression job so KARPENTER_CORE_DIR and TEST_SUITE are passed as environment
assignments on the same command invocation, ensuring the script receives both
regression settings.
- Around line 79-83: Remove the unused yq setup commands from the job script:
delete the /tmp/karpenter directory creation, YQ assignment, curl download,
chmod, and PATH export. Leave the surrounding upstream-e2e.sh execution
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 40f7e9eb-3009-4bf0-bd40-8e7b094bda2d
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/aws-karpenter-provider-aws/openshift-aws-karpenter-provider-aws-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/aws-karpenter-provider-aws/openshift-aws-karpenter-provider-aws-main__periodics.yaml
60f202b to
721a224
Compare
721a224 to
63829bf
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/aws-karpenter-provider-aws/openshift-aws-karpenter-provider-aws-main__periodics.yaml`:
- Around line 91-95: Restore provisioning of the system yq binary before
invoking the karpenter-upstream-e2e target in the hypershift clone flow,
ensuring it is available on PATH when adjust-ec2nodeclass.sh runs. Keep the
existing KARPENTER_CORE_DIR, TEST_SUITE, and make invocation unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 618b6aa3-b248-4e70-b631-704c80cac34f
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/aws-karpenter-provider-aws/openshift-aws-karpenter-provider-aws-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/aws-karpenter-provider-aws/openshift-aws-karpenter-provider-aws-main__periodics.yaml
998bd82 to
a1f5bb7
Compare
|
/pj-rehearse periodic-ci-openshift-aws-karpenter-provider-aws-main-periodics-e2e-aws-karpenter-core-regression |
|
@fishereskew: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
It's been a while and multiple rebases since we came back to this so unfortunately there's some hiccups here 😓.
I see two failures:
STEP: waiting for initialized nodes to be == to 1 - /tmp/tmp.qAuW0GwCgQ/test/suites/regression/nodeclaim_test.go:70 @ 07/29/26 20:49:06.641
[CREATED/UPDATED 2026-07-29T20:49:06Z] NODECLAIM carpthunder-3-sc0uuwenw3 ready=false launched=false registered=false initialized=false
[CREATED/UPDATED 2026-07-29T20:49:11Z] NODECLAIM carpthunder-3-sc0uuwenw3 ready=false launched=true registered=false initialized=false
[CREATED/UPDATED 2026-07-29T21:01:02Z] POD openshift-insights/periodic-gathering-nsr9l-klksx provisionable=false phase=Pending nodename= owner=[]v1.OwnerReference{v1.OwnerReference{APIVersion:"batch/v1", Kind:"Job", Name:"periodic-gathering-nsr9l", UID:"6dce900f-c5f9-4c1c-896a-562cecff4906", Controller:(*bool)(0xc0010aa1b0), BlockOwnerDeletion:(*bool)(0xc0010aa1b1)}} []
[CREATED/UPDATED 2026-07-29T21:01:02Z] POD openshift-insights/periodic-gathering-nsr9l-klksx provisionable=true phase=Pending nodename= owner=[]v1.OwnerReference{v1.OwnerReference{APIVersion:"batch/v1", Kind:"Job", Name:"periodic-gathering-nsr9l", UID:"6dce900f-c5f9-4c1c-896a-562cecff4906", Controller:(*bool)(0xc000daa3e0), BlockOwnerDeletion:(*bool)(0xc000daa3e1)}} []
[FAILED] Timed out after 900.001s.
The function passed to Eventually failed at /tmp/tmp.qAuW0GwCgQ/test/pkg/environment/common/expectations.go:826 with:
Expected
<int>: 0
to be ==
Seems like creating a standalone NodeClaim does not work on our platform, and we should probably figure out why. I would try running this locally to see the steps in real time and looking at events, status fields, and logs of karpenter-operator and karpenter in the management cluster. I know you are sort of running out of time so, this is entirely up to you on whether you want to try to muscle your way through this.
The second problem is that the tests assume NodeOverlay to be in the cluster, but right now we don't deploy it in HyperShift.
------- END CONTROLLER LOGS -------
[FAILED] Timed out after 900.001s.
The function passed to Eventually failed at /tmp/tmp.qAuW0GwCgQ/test/pkg/environment/common/setup.go:176 with:
Expected success, but got an error:
<*meta.NoKindMatchError | 0xc00170da00>:
no matches for kind "NodeOverlay" in version "karpenter.sh/v1alpha1"
{
GroupKind: {
Group: "karpenter.sh",
Kind: "NodeOverlay",
},
SearchedVersions: ["v1alpha1"],
}
In [AfterEach] at: /tmp/tmp.qAuW0GwCgQ/test/pkg/environment/common/setup.go:189 @ 07/29/26 21:19:07.833
< Exit [AfterEach] TOP-LEVEL - /tmp/tmp.qAuW0GwCgQ/test/suites/regression/suite_test.go:80 @ 07/29/26 21:19:07.833 (15m1.185s)
• [FAILED] [1801.531 seconds]
So either we need to start deploying NodeOverlay, or have a carry patch which ignores that CRD in our kubernetes-sigs/karpenter fork. Since we need to eventually enable NodeOverlay anyways, we should probably just start allowing the karpenter-operator to deploy it to the guest cluster: https://github.com/openshift/hypershift/blob/1c9f18937db3360121ffd41acad4ed333364afec/karpenter-operator/controllers/karpenter/karpenter_controller.go#L58-L62
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
openshift-ci
Bot
added
the
needs-rebase
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: fishereskew The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@fishereskew, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
c2b3f98 to
d23e9e5
Compare
openshift-ci
Bot
removed
the
needs-rebase
…ator Signed-off-by: Fisher Eskew <feskew@redhat.com> add: weekly karpenter core regression e2e periodic for karpenter-operator Signed-off-by: Fisher Eskew <feskew@redhat.com>
d23e9e5 to
09c69b0
Compare
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
fishereskew
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@fishereskew: This pull request references AUTOSCALE-166 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/pj-rehearse periodic-ci-openshift-karpenter-operator-main-periodics-e2e-karpenter-core-regression |
|
@shyadav10: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@fishereskew: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/retest |
4 participants
jira: AUTOSCALE-166
Adds a weekly periodic job for aws-karpenter-provider-aws deployed via Hypershift Autonode to run the upstream karpenter core Regression suite. The job clones Karpenter Core for the regression suite and Hypershift to run the upstream-e2e.sh script for test setup and running. It's set to run at 6:00am on Saturdays.
This was previously added to Hypershift in #62456 and removed in #68109. It is being added here to attribute failures to aws-karpenter-provider-aws instead of polluting Hypershift's tests.
Update: Moved the periodic job from the aws-karpenter-provider-aws CI config to the karpenter-operator CI config. The prow job now calls make karpenter-core-regression in the karpenter-operator repo, where the test scripts and configuration live so the team can update them without changes to the release repo. See openshift/karpenter-operator#20 for the make target and scripts, and openshift/hypershift#9262 for the NodeOverlay CRD wiring needed for the tests to pass.
Summary by CodeRabbit
aws-karpenter-provider-aws.aws-karpenter-provider-aws.