Skip to content

LCORE-2035- Add TLS fixes for konflux run #1929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
radofuchs merged 8 commits into from
Jun 17, 2026
Merged

LCORE-2035- Add TLS fixes for konflux run #1929

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c36d758
Add TLS fixes for konflux run
May 20, 2026
6a27f64
reorder the new test to get the best out of the test setup
Jun 15, 2026
3f326cc
extend waits
Jun 15, 2026
c2ff87a
fix logic
Jun 16, 2026
66e7bf4
fix
Jun 16, 2026
68521e7
move tls to the start of test run to prevent infrastructure issues
Jun 16, 2026
d874403
revert all unwanted changes
Jun 17, 2026
d2d38c4
disable tls tests
Jun 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .tekton/integration-tests/pipeline/lightspeed-stack-integration-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,8 @@ spec:
echo "========== End parameters =========="
- name: lightspeed-stack-integration-tests
description: Task to run integration tests from lightspeed-stack repository
# Full Behave suite (proxy + tls) can exceed 2h; needs PipelineRun timeouts >= this value.
timeout: 3h
params:
- name: SNAPSHOT
value: $(params.SNAPSHOT)
Expand Down
104 changes: 104 additions & 0 deletions tests/e2e-prow/rhoai/manifests/lightspeed/e2e-mock-tls-inference.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# Mock HTTPS OpenAI API for tls-*.feature (Konflux / Prow; no Docker Compose).
# Llama Stack run.yaml uses https://e2e-mock-tls-inference.<ns>.svc.cluster.local:8443|8444|8445/v1
apiVersion: v1
kind: Pod
metadata:
name: e2e-mock-tls-inference
labels:
app: e2e-mock-tls-inference
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: e2e-mock-tls-inference
image: python:3.12-slim
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: PYTHONPATH
value: /app:/tmp/pydeps
command:
- /bin/sh
- -c
- |
set -e
pip install --quiet --no-cache-dir --target /tmp/pydeps 'trustme>=1.2.1' 'cryptography>=42.0.0'

@coderabbitai coderabbitai Bot Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, find and read the file
find . -name "e2e-mock-tls-inference.yaml" -type f

Repository: lightspeed-core/lightspeed-stack

Length of output: 148

🏁 Script executed:

# Check if the file exists and read around line 37
if [ -f "tests/e2e-prow/rhoai/manifests/lightspeed/e2e-mock-tls-inference.yaml" ]; then
  echo "=== File content with line numbers ===" 
  cat -n "tests/e2e-prow/rhoai/manifests/lightspeed/e2e-mock-tls-inference.yaml" | sed -n '30,45p'
else
  echo "File not found at expected path"
fi

Repository: lightspeed-core/lightspeed-stack

Length of output: 808

🌐 Web query:

trustme python package latest version 2024

💡 Result:

The latest version of the trustme Python package released in 2024 was version 1.2.0, which was published on October 7, 2024 [1][2]. As of today, June 15, 2026, the project has since released version 1.2.1 (published January 2, 2025) [1][2][3].

Citations:

🌐 Web query:

cryptography python package version 42 43 release

💡 Result:

The cryptography Python package releases for versions 42 and 43 are summarized below based on their official changelogs. Version 42 Series The 42.x series began with version 42.0.0, released on January 22, 2024 [1][2]. The series concluded with version 42.0.8, released on June 4, 2024 [1][3]. - 42.0.0: 2024-01-22 - 42.0.1: 2024-01-24 - 42.0.2: 2024-01-30 - 42.0.3: 2024-02-15 - 42.0.4: 2024-02-20 - 42.0.5: 2024-02-23 - 42.0.6: 2024-05-04 - 42.0.7: 2024-05-06 - 42.0.8: 2024-06-04 Version 43 Series The 43.x series began with version 43.0.0, released on July 20, 2024 [1][2]. Major changes in 43.0.0 included dropping support for OpenSSL versions older than 1.1.1e and LibreSSL versions older than 3.8 [2][4]. - 43.0.0: 2024-07-20 - 43.0.1: 2024-09-03 - 43.0.2: 2024-10-18 - 43.0.3: 2024-10-18 [1][2]

Citations:

Pin runtime Python dependencies to deterministic version ranges.

Line 37 installs trustme and cryptography with open-ended >= constraints, causing non-deterministic e2e behavior when upstream releases new versions. The suggested patch adds upper bounds, though the cryptography<43 bound is outdated since version 43.0.0 was released in July 2024 and multiple 43.x versions exist as of today. Update the constraint to reflect currently compatible versions (e.g., cryptography>=42.0.0,<44 if 43.x is compatible, or pin to the tested version explicitly).

Suggested patch 
-          pip install --quiet --no-cache-dir --target /tmp/pydeps 'trustme>=1.2.1' 'cryptography>=42.0.0'
+          pip install --quiet --no-cache-dir --target /tmp/pydeps \
+            'trustme>=1.2.1,<2' \
+            'cryptography>=42.0.0,<44'
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
pip install --quiet --no-cache-dir --target /tmp/pydeps 'trustme>=1.2.1' 'cryptography>=42.0.0'
pip install --quiet --no-cache-dir --target /tmp/pydeps \
'trustme>=1.2.1,<2' \
'cryptography>=42.0.0,<44'
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e-prow/rhoai/manifests/lightspeed/e2e-mock-tls-inference.yaml` at
line 37, The pip install command on line 37 uses open-ended version constraints
(trustme>=1.2.1 and cryptography>=42.0.0) without upper bounds, causing
non-deterministic behavior when upstream releases new versions. Add upper bound
constraints to both packages to make the dependency versions deterministic. For
trustme, add an upper bound like >=1.2.1,<2 (or appropriate upper bound based on
tested compatibility). For cryptography, update the constraint from >=42.0.0 to
a bounded range reflecting currently compatible versions such as >=42.0.0,<44
(or pin to the specific tested version if 43.x versions have incompatibilities
with your code). Ensure both packages have explicit upper bounds to prevent
non-deterministic e2e test behavior.

NS="${POD_NAMESPACE:-default}"
export TLS_CERT_DNS_NAMES="mock-tls-inference,localhost,127.0.0.1,e2e-mock-tls-inference,e2e-mock-tls-inference.${NS}.svc.cluster.local"
exec python /app/server.py
ports:
- containerPort: 8443
name: tls
- containerPort: 8444
name: mtls
- containerPort: 8445
name: mismatch
volumeMounts:
- name: server-script
mountPath: /app/server.py
subPath: server.py
readOnly: true
- name: certs-work
mountPath: /certs
readinessProbe:
exec:
command:
- python3
- -c
- |
import ssl, urllib.request
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
urllib.request.urlopen("https://localhost:8443/health", context=ctx)
initialDelaySeconds: 8
periodSeconds: 5
livenessProbe:
exec:
command:
- python3
- -c
- |
import ssl, urllib.request
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
urllib.request.urlopen("https://localhost:8443/health", context=ctx)
initialDelaySeconds: 15
periodSeconds: 20
volumes:
- name: server-script
configMap:
name: e2e-mock-tls-inference-script
- name: certs-work
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: e2e-mock-tls-inference
spec:
selector:
app: e2e-mock-tls-inference
ports:
- name: tls
port: 8443
targetPort: tls
- name: mtls
port: 8444
targetPort: mtls
- name: mismatch
port: 8445
targetPort: mismatch
33 changes: 32 additions & 1 deletion tests/e2e-prow/rhoai/manifests/lightspeed/llama-stack-openai.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,28 @@ spec:
- -c
- |
set -e
# Fast-path: PVC already has a valid venv from a previous pod creation in this pipeline run.
# TLS scenarios delete+recreate this pod up to 16 times; skipping the expensive install
# reduces per-restart time from ~6-15 min to ~30-90 s (just RAG seed refresh + chown).
if [[ -d /opt/app-root/.venv ]] \
&& /opt/app-root/.venv/bin/python --version >/dev/null 2>&1 \
&& [[ -d /opt/app-root/src ]]; then
echo "PVC cache hit: app-root already provisioned — skipping full install"
Comment on lines +44 to +47

@coderabbitai coderabbitai Bot Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fast-path cache validation misses a required startup artifact.

Line 44-47 treats cache as valid without checking /opt/app-root/llama_stack_configuration.py, but Line 182 executes it unconditionally. A partial PVC cache can cause repeat crash loops.

Suggested patch 
-          if [[ -d /opt/app-root/.venv ]] \
-              && /opt/app-root/.venv/bin/python --version >/dev/null 2>&1 \
-              && [[ -d /opt/app-root/src ]]; then
+          if [[ -d /opt/app-root/.venv ]] \
+              && /opt/app-root/.venv/bin/python --version >/dev/null 2>&1 \
+              && [[ -d /opt/app-root/src ]] \
+              && [[ -f /opt/app-root/llama_stack_configuration.py ]]; then

Also applies to: 182-185

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e-prow/rhoai/manifests/lightspeed/llama-stack-openai.yaml` around
lines 44 - 47, The cache validation logic fails to verify the existence of a
critical startup artifact, allowing partial PVC caches to be considered valid.
In the conditional block that checks for PVC cache hits (the if statement
checking for /opt/app-root/.venv directory and Python version), add an
additional check to ensure /opt/app-root/llama_stack_configuration.py exists
before marking the cache as valid. This required artifact is executed
unconditionally later in the script, so the cache should only be considered a
hit when this file is present.

mkdir -p /opt/app-root/.e2e-rag-seed /opt/app-root/src/.llama/storage/rag /opt/app-root/src/.llama/storage/files
if [[ -f /rag-seed/kv_store.db.gz ]]; then
gzip -dc /rag-seed/kv_store.db.gz > /opt/app-root/.e2e-rag-seed/kv_store.db
_sz=$(stat -c%s /opt/app-root/.e2e-rag-seed/kv_store.db)
if [[ "${_sz}" -lt 1048576 ]]; then
echo "FATAL: RAG seed too small (${_sz} bytes); check rag-data ConfigMap"
exit 1
fi
cp -f /opt/app-root/.e2e-rag-seed/kv_store.db /opt/app-root/src/.llama/storage/rag/kv_store.db
fi
chmod -R 775 /opt/app-root && chown -R 1001:0 /opt/app-root
echo "PVC fast-path complete"
exit 0
fi
# Full provisioning (PVC is empty — first pod creation this pipeline run).
REPO_URL="${REPO_URL:-https://github.com/lightspeed-core/lightspeed-stack.git}"
REPO_REVISION="${REPO_REVISION:-main}"
case "$REPO_URL" in git@github.com:*) REPO_URL="https://github.com/${REPO_URL#git@github.com:}"; esac
Expand Down Expand Up @@ -201,9 +223,14 @@ spec:
mountPath: /tmp/interception-proxy-ca.pem
subPath: ca.pem
readOnly: true
# tls-*.feature: client/CA PEMs from Secret e2e-mock-tls-certs (optional).
- name: mock-tls-certs
mountPath: /certs
readOnly: true
volumes:
- name: app-root
emptyDir: {}
persistentVolumeClaim:
claimName: llama-stack-app-root
- name: config-cm
configMap:
name: llama-stack-config
Expand All @@ -217,3 +244,7 @@ spec:
secret:
secretName: e2e-interception-proxy-ca
optional: true
- name: mock-tls-certs
secret:
secretName: e2e-mock-tls-certs
optional: true
20 changes: 20 additions & 0 deletions tests/e2e-prow/rhoai/pipeline-konflux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,25 @@ log "✅ Mock servers deployed"
#========================================
progress "Deploying lightspeed-stack and llama-stack"

# PVC for llama-stack app-root: caches dnf/uv/git install so TLS per-scenario pod
# recreates skip the expensive init (~6-15 min → ~1-2 min). Delete first to guarantee
# a fresh checkout for this pipeline revision; re-create immediately so the pod can bind.
log "Recreating llama-stack-app-root PVC (fresh per pipeline run)..."
oc delete pvc llama-stack-app-root -n "$NAMESPACE" --ignore-not-found=true 2>/dev/null || true
cat <<'EOF' | oc apply -n "$NAMESPACE" -f -
apiVersion: v1
Comment on lines +150 to +152

@coderabbitai coderabbitai Bot Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

PVC delete/recreate sequence has a race.

Line 150 deletes the claim and Line 151 reapplies immediately. PVC deletion is asynchronous; recreate can fail while the old claim is still terminating.

Suggested patch 
 oc delete pvc llama-stack-app-root -n "$NAMESPACE" --ignore-not-found=true 2>/dev/null || true
+for _ in $(seq 1 60); do
+  oc get pvc llama-stack-app-root -n "$NAMESPACE" >/dev/null 2>&1 || break
+  sleep 2
+done
+oc get pvc llama-stack-app-root -n "$NAMESPACE" >/dev/null 2>&1 && {
+  echo "❌ PVC llama-stack-app-root is still terminating"
+  exit 1
+}
 cat <<'EOF' | oc apply -n "$NAMESPACE" -f -
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
oc delete pvc llama-stack-app-root -n "$NAMESPACE" --ignore-not-found=true 2>/dev/null || true
cat <<'EOF' | oc apply -n "$NAMESPACE" -f -
apiVersion: v1
oc delete pvc llama-stack-app-root -n "$NAMESPACE" --ignore-not-found=true 2>/dev/null || true
for _ in $(seq 1 60); do
oc get pvc llama-stack-app-root -n "$NAMESPACE" >/dev/null 2>&1 || break
sleep 2
done
oc get pvc llama-stack-app-root -n "$NAMESPACE" >/dev/null 2>&1 && {
echo "❌ PVC llama-stack-app-root is still terminating"
exit 1
}
cat <<'EOF' | oc apply -n "$NAMESPACE" -f -
apiVersion: v1
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e-prow/rhoai/pipeline-konflux.sh` around lines 150 - 152, The script
deletes the PVC with oc delete on line 150 and immediately reapplies it on line
151, creating a race condition since PVC deletion is asynchronous. Add a wait
mechanism after the oc delete pvc llama-stack-app-root command to ensure the PVC
is fully removed before attempting to recreate it with oc apply. Use an oc wait
command or retry logic to verify the PVC is completely deleted and gone from the
namespace before proceeding with the reapplication.

kind: PersistentVolumeClaim
metadata:
name: llama-stack-app-root
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
EOF
log "✅ llama-stack-app-root PVC created"

# Llama run config: single source with GitHub E2E (tests/e2e/configs/run-ci.yaml).
# Lightspeed stack: same tree as local/docker E2E (tests/e2e/configuration/server-mode).
oc create configmap llama-stack-config -n "$NAMESPACE" \
Expand Down Expand Up @@ -393,6 +412,7 @@ if [[ -n "${E2E_LLAMA_PORT_FORWARD_PID_FILE:-}" && -f "$E2E_LLAMA_PORT_FORWARD_P
fi
rm -f "$E2E_LLAMA_PORT_FORWARD_PID_FILE"
fi

kill $PF_LCS_PID 2>/dev/null || true
kill $PF_JWKS_PID 2>/dev/null || true
kill $PF_LLAMA_PID 2>/dev/null || true
Expand Down
18 changes: 17 additions & 1 deletion tests/e2e-prow/rhoai/pipeline-services-konflux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,25 @@ oc create secret generic llama-stack-ip-secret \
-n "$NAMESPACE" \
--dry-run=client -o yaml | oc apply -f -

# PVC must exist before the pod (pipeline-konflux.sh creates it; guard here for standalone use).
oc get pvc llama-stack-app-root -n "$NAMESPACE" >/dev/null 2>&1 || \
oc apply -n "$NAMESPACE" -f - <<'PVCEOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: llama-stack-app-root
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
PVCEOF

timeout 120 oc delete pod llama-stack-service -n "$NAMESPACE" --ignore-not-found=true --wait=true 2>/dev/null || true
oc apply -n "$NAMESPACE" -f "$BASE_DIR/manifests/lightspeed/llama-stack-openai.yaml"
oc wait pod/llama-stack-service -n "$NAMESPACE" --for=condition=Ready --timeout=600s
# First boot runs the full init (dnf + git clone + uv sync ≈ 6-15 min); use a generous timeout.
oc wait pod/llama-stack-service -n "$NAMESPACE" --for=condition=Ready --timeout=900s
oc label pod llama-stack-service pod=llama-stack-service -n "$NAMESPACE"
oc expose pod llama-stack-service --name=llama-stack-service-svc --port=8321 --type=ClusterIP -n "$NAMESPACE"

Expand Down
Loading
Loading