cleaning up any prow related reference, updated e2e testing doc on how to run okp tests locally

Author: Sofiya Nuryyeva

.tekton/integration-tests/pipeline/lightspeed-stack-integration-test.yaml

@@ -281,11 +281,6 @@ spec:
                 value: "true"
               - name: OTEL_ANONYMIZATION_SECRET
                 value: "lightspeed-stack-otel-anonymization-dev-default"
-              # PipelineRun metadata for ownerReference in secret creation
-              - name: TEKTON_PIPELINERUN_NAME
-                value: "$(context.pipelineRun.name)"
-              - name: TEKTON_PIPELINERUN_UID
-                value: "$(context.pipelineRun.uid)"
             image: registry.access.redhat.com/ubi9/ubi-minimal
             script: |
               set +e

docs/testing/e2e_scenarios.md

@@ -115,6 +115,20 @@
 * Check if models can be filtered
 * Check if filtering can return empty list of models
 
+## [`okp_rag.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/okp_rag.feature)
+
+Konflux only (`@konflux-only`). Local stack: [Running OKP RAG tests locally](e2e_testing.md#running-okp-rag-tests-locally).
+
+* Offline mode query with inline RAG returns rag_chunks and referenced_documents
+* Online mode streaming query with inline RAG returns referenced_documents
+* Query with inline RAG with dynamic semantic filter returns rag_chunks and referenced_documents
+* Offline query API with OKP tool RAG has rag_chunk and referenced_documents returned
+* Online responses API with OKP tool RAG has rag results returned
+* Query succeeds with empty rag_chunks when OKP server is unavailable
+* Streaming query succeeds with empty referenced_documents when OKP server is unavailable
+* Query returns no rag_chunks and no reference_documents when OKP is disabled
+* Streaming query returns no referenced_documents when OKP is disabled
+
 ## [`proxy.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/proxy.feature)
 
 * LLM traffic is routed through a configured tunnel proxy

docs/testing/e2e_testing.md

@@ -9,14 +9,15 @@ This guide describes how to run, extend, and understand the Lightspeed Core Stac
 1. [Overview](#overview)
 2. [Directory Layout](#directory-layout)
 3. [How to Run E2E Tests](#how-to-run-e2e-tests)
-4. [Environment Variables](#environment-variables)
-5. [Deployment Modes: Server vs Library](#deployment-modes-server-vs-library)
-6. [Tags and Hooks](#tags-and-hooks)
-7. [Configuration Files](#configuration-files)
-8. [Feature Files and Steps](#feature-files-and-steps)
-9. [Gherkin Keywords in Feature Files](#gherkin-keywords-in-feature-files)
-10. [Writing New Scenarios](#writing-new-scenarios)
-11. [Troubleshooting](#troubleshooting)
+4. [Running OKP RAG tests locally](#running-okp-rag-tests-locally)
+5. [Environment Variables](#environment-variables)
+6. [Deployment Modes: Server vs Library](#deployment-modes-server-vs-library)
+7. [Tags and Hooks](#tags-and-hooks)
+8. [Configuration Files](#configuration-files)
+9. [Feature Files and Steps](#feature-files-and-steps)
+10. [Gherkin Keywords in Feature Files](#gherkin-keywords-in-feature-files)
+11. [Writing New Scenarios](#writing-new-scenarios)
+12. [Troubleshooting](#troubleshooting)
 
 ---
 
@@ -135,6 +136,35 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
 
 ---
 
+## Running OKP RAG tests locally
+
+`okp_rag.feature` is `@konflux-only` (`make test-e2e` skips it). Locally: OKP in Docker, OGX and LCS as host processes. Needs `registry.redhat.io` login, `OPENAI_API_KEY`, and `../lightspeed-providers`. Do not set `E2E_KONFLUX_E2E=1` (that deploys OKP on Kubernetes).
+
+```bash
+docker login registry.redhat.io
+docker run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
+
+export RH_SERVER_OKP=http://localhost:8081/solr
+export PYTHONPATH="$(cd ../lightspeed-providers && pwd)${PYTHONPATH:+:$PYTHONPATH}"
+export EXTERNAL_PROVIDERS_DIR="$(cd ../lightspeed-providers && pwd)/resources/external_providers"
+OKP_LCS=tests/e2e/configuration/server-mode/lightspeed-stack-okp-offline.yaml
+
+# terminal 1 — OGX :8321
+uv run python src/ogx_configuration.py -c "$OKP_LCS" -i run.yaml -o run_enriched.yaml
+uv run ogx stack run run_enriched.yaml --port 8321
+
+# terminal 2 — LCS :8080
+make run-ogx CONFIG="$OKP_LCS"
+
+curl -sX POST http://localhost:8080/v1/query \
+  -H "Content-Type: application/json" \
+  -d '{"query": "configure remote desktop using gnome"}' | jq .
+```
+
+Confirm OKP at http://localhost:8081. Other `lightspeed-stack-okp-*.yaml` files cover tool RAG / online URLs (re-enrich and restart OGX). Product setup: [OKP guide](../user_doc/okp_guide.md).
+
+---
+
 ## Environment Variables
 
 | Variable                        | Default     | Description                                                                                                                               |
@@ -150,6 +180,8 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
 | `E2E_DEFAULT_PROVIDER_OVERRIDE` | —           | Override default provider id (e.g. `openai`).                                                                                             |
 | `FAISS_VECTOR_STORE_ID`         | —           | Vector store id for FAISS-related scenarios.                                                                                              |
 | `RUNNING_PROW`                  | —           | Set in Prow/OpenShift; enables Prow config paths and pod/container ops.                                                                   |
+| `E2E_KONFLUX_E2E`               | —           | `1` in Konflux only. Unskips `@konflux-only` and deploys OKP as a pod.                                                                     |
+| `RH_SERVER_OKP`                 | —           | OKP/Solr URL (local default `http://localhost:8081/solr`).                                                                                |
 | `OPENAI_API_KEY`                | —           | **Required.** Used by the app and OGX for LLM calls (e.g. OpenAI). The E2E tests and the stack will not run correctly without it. |
 
 
@@ -175,6 +207,8 @@ All tag behaviour is implemented in **`features/environment.py`**: the hooks (`b
 | Tag                             | Effect                                                                                                                                                  |
 |---------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `@skip`                         | Scenario is skipped (reason: "Marked with @skip"). Use for broken or WIP scenarios.                                                                     |
+| `@konflux-only`                 | Skipped unless `E2E_KONFLUX_E2E=1`. Used by `okp_rag.feature`.                                                                           |
+| `@cfg_okp`                      | OKP Solr RAG. Konflux deploys OKP in `before_feature`. See [Running OKP RAG tests locally](#running-okp-rag-tests-locally).              |
 | `@skip-in-library-mode`         | Scenario is skipped when `E2E_DEPLOYMENT_MODE=library`. Used for tests that require a separate OGX (e.g. connection disruption).                |
 | `@local`                        | Skipped unless running in "local" mode (context flag).                                                                                                  |
 | `@InvalidFeedbackStorageConfig` | Before scenario: switch to invalid-feedback-storage config and restart container. After: restore feature config and restart.                            |
@@ -245,6 +279,7 @@ The feature files below are run in the order given in `tests/e2e/test_list.txt`:
 | `rest_api.feature`               | REST API: OpenAPI endpoint.                                                                                                             |
 | `mcp.feature`                    | MCP (Model Context Protocol): tools, query, streaming_query with MCP auth (required, token, invalid token).                             |
 | `models.feature`                 | Models endpoint: list models, filter, empty result; error when OGX unreachable.                                                 |
+| `okp_rag.feature`                | OKP Solr RAG (`@konflux-only`). Local stack: [Running OKP RAG tests locally](#running-okp-rag-tests-locally).                           |
 
 
 If you add a new feature file, add it to **`tests/e2e/test_list.txt`** so it is included when you run the full E2E suite (e.g. `make test-e2e`). The order in that file is the run order.
@@ -354,6 +389,6 @@ Here, **Given** sets state, **When** performs the HTTP call, **Then** and **And*
 - **Readonly database (SQLite) in OGX**: If the RAG KV DB is on a bind-mounted path that becomes read-only (e.g. after restart), move it to a named volume (e.g. via `KV_RAG_PATH` in docker-compose) so writes succeed.
 - **ChunkedEncodingError on streaming_query**: The step for streaming_query uses `stream=True` and consumes the stream; if you add new streaming steps, avoid reading the full response with `response.content` and use the same stream-reading pattern so a server close after an error event does not raise.
 - **Event loop is closed (httpx/AsyncClient)**: In E2E, any code that creates an `AsyncOgxClient` (e.g. for shields) must close it (e.g. `await client.close()`) in a `finally` block before the event loop is torn down (e.g. before `asyncio.run()` returns).
-- **Scenarios skipped**: Check tags (`@skip`, `@skip-in-library-mode`, `@local`) and `E2E_DEPLOYMENT_MODE`; ensure the scenario is not excluded by `--tags=-skip` (or the opposite if you intend to run only skipped scenarios for debugging).
+- **Scenarios skipped**: Check tags (`@skip`, `@skip-in-library-mode`, `@local`, `@konflux-only`) and `E2E_DEPLOYMENT_MODE`; ensure the scenario is not excluded by `--tags=-skip` (or the opposite if you intend to run only skipped scenarios for debugging).
 
 For more on test structure and commands, see the main project guide (`CLAUDE.md`) and `tests/e2e/features/steps/README.md`.

docs/user_doc/okp_guide.md

@@ -54,12 +54,14 @@ validating that queries return referenced chunks.
 Start the OKP RAG service with Podman or Docker:
 
 **Using Podman:**
+
 ```bash
 podman login registry.redhat.io
 podman run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
 ```
 
 **Using Docker:**
+
 ```bash
 docker login registry.redhat.io
 docker run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
@@ -349,4 +351,6 @@ If you see no RAG context, verify:
 1. OKP is up at http://localhost:8081
 2. `lightspeed-stack.yaml` has `okp` under `rag.inline` and/or `rag.tool` as in Step 4
 
+The Behave suite `okp_rag.feature` (`@cfg_okp`) runs in Konflux only. To stand up the same OKP + OGX + LCS topology on a laptop (OKP in Docker, OGX and LCS as host processes), see [Running OKP RAG tests locally](../testing/e2e_testing.md#running-okp-rag-tests-locally).
+
 ---

tests/e2e-prow/rhoai/manifests/lightspeed/ogx-prow.yaml

@@ -84,9 +84,7 @@ spec:
       workingDir: /opt/app-root
       env:
         - name: PYTHONPATH
-          value: "/opt/app-root:/opt/app-root/src"
-        - name: EXTERNAL_PROVIDERS_DIR
-          value: "/opt/app-root/providers.d"
+          value: "/opt/app-root/src"
         - name: HOME
           value: "/opt/app-root/src"
         # Prevent OGX from shutil.move(~/.llama → ~/.ogx) which would steal the fixture.

tests/e2e-prow/rhoai/pipeline-konflux.sh

@@ -110,106 +110,42 @@ oc create secret docker-registry quay-lightspeed-pull-secret \
 # Link the secret to default service account for image pulls
 oc secrets link default quay-lightspeed-pull-secret --for=pull -n "$NAMESPACE" 2>/dev/null || echo "⚠️  Secret already linked to default SA"
 
-# Create Red Hat registry pull secret for OKP images
-# Option 1: Use mounted docker-registry secret (preferred - simpler)
+# Create Red Hat registry pull secret for OKP images in the ephemeral test
+# namespace. Do not set ownerReferences to the Konflux PipelineRun: that object
+# lives on a different cluster, and the ephemeral GC would treat the Secret as
+# orphaned. Namespace teardown deletes the Secret with everything else.
 if [[ -f /var/run/redhat-registry-pull-secret/.dockerconfigjson ]]; then
   log "Creating Red Hat registry pull secret from mounted docker-registry secret..."
+  oc create secret generic redhat-registry-pull-secret \
+    --from-file=.dockerconfigjson=/var/run/redhat-registry-pull-secret/.dockerconfigjson \
+    --type=kubernetes.io/dockerconfigjson \
+    -n "$NAMESPACE" \
+    --dry-run=client -o yaml | oc apply -f -
+  log "✅ Red Hat registry pull secret created"
+  oc secrets link default redhat-registry-pull-secret --for=pull -n "$NAMESPACE" 2>/dev/null || echo "⚠️  Secret already linked to default SA"
 
-  DOCKERCONFIG_BASE64=$(cat /var/run/redhat-registry-pull-secret/.dockerconfigjson | base64 -w0)
-
-  # Use PipelineRun metadata for ownerReference (provided by Tekton context)
-  # This ensures automatic cleanup when the PipelineRun completes
-  if [[ -n "${TEKTON_PIPELINERUN_NAME:-}" && -n "${TEKTON_PIPELINERUN_UID:-}" ]]; then
-    log "Setting ownerReference to PipelineRun: $TEKTON_PIPELINERUN_NAME (UID: ${TEKTON_PIPELINERUN_UID:0:8}...)"
-
-    # Create secret with ownerReference using YAML (ensures automatic cleanup)
-    cat <<EOF | oc apply -f -
-apiVersion: v1
-kind: Secret
-metadata:
-  name: redhat-registry-pull-secret
-  namespace: $NAMESPACE
-  ownerReferences:
-  - apiVersion: tekton.dev/v1beta1
-    kind: PipelineRun
-    name: $TEKTON_PIPELINERUN_NAME
-    uid: $TEKTON_PIPELINERUN_UID
-    controller: false
-    blockOwnerDeletion: false
-type: kubernetes.io/dockerconfigjson
-data:
-  .dockerconfigjson: $DOCKERCONFIG_BASE64
-EOF
-    log "✅ Red Hat registry pull secret created with ownerReference"
-
-    # Link to default service account
-    oc secrets link default redhat-registry-pull-secret --for=pull -n "$NAMESPACE" 2>/dev/null || echo "⚠️  Secret already linked to default SA"
-  else
-    # Fallback: create without ownerReference (requires manual cleanup)
-    log "⚠️  TEKTON_PIPELINERUN_NAME/UID not set - creating secret without ownerReference"
-    log "⚠️  Manual cleanup required after test completion"
-
-    cat <<EOF | oc apply -f -
-apiVersion: v1
-kind: Secret
-metadata:
-  name: redhat-registry-pull-secret
-  namespace: $NAMESPACE
-type: kubernetes.io/dockerconfigjson
-data:
-  .dockerconfigjson: $DOCKERCONFIG_BASE64
-EOF
-    log "✅ Red Hat registry pull secret created (without ownerReference)"
-    oc secrets link default redhat-registry-pull-secret --for=pull -n "$NAMESPACE" 2>/dev/null || echo "⚠️  Secret already linked to default SA"
-  fi
-
-# Option 2: Fallback to username/password mounted separately (legacy approach)
 elif [[ -d /var/run/redhat-registry-username ]] && [[ -d /var/run/redhat-registry-password ]]; then
   log "Creating Red Hat registry pull secret from username/password..."
   REDHAT_USERNAME=""
   REDHAT_PASSWORD=""
 
-  # Read username
   shopt -s nullglob
   for _f in /var/run/redhat-registry-username/*; do
     [[ -f "$_f" ]] && REDHAT_USERNAME="$(cat "$_f")" && break
   done
-
-  # Read password
   for _f in /var/run/redhat-registry-password/*; do
     [[ -f "$_f" ]] && REDHAT_PASSWORD="$(cat "$_f")" && break
   done
   shopt -u nullglob
 
   if [[ -n "$REDHAT_USERNAME" ]] && [[ -n "$REDHAT_PASSWORD" ]]; then
-    # Use PipelineRun metadata for ownerReference (provided by Tekton context)
-    if [[ -n "${TEKTON_PIPELINERUN_NAME:-}" && -n "${TEKTON_PIPELINERUN_UID:-}" ]]; then
-      log "Setting ownerReference to PipelineRun: $TEKTON_PIPELINERUN_NAME (UID: ${TEKTON_PIPELINERUN_UID:0:8}...)"
-
-      # Create secret with ownerReference (oc handles JSON encoding safely)
-      oc create secret docker-registry redhat-registry-pull-secret \
-        --docker-server=registry.redhat.io \
-        --docker-username="$REDHAT_USERNAME" \
-        --docker-password="$REDHAT_PASSWORD" \
-        -n "$NAMESPACE" \
-        --dry-run=client -o json | \
-      jq --arg name "$TEKTON_PIPELINERUN_NAME" --arg uid "$TEKTON_PIPELINERUN_UID" \
-        '.metadata.ownerReferences = [{"apiVersion":"tekton.dev/v1beta1","kind":"PipelineRun","name":$name,"uid":$uid,"controller":false,"blockOwnerDeletion":false}]' | \
-      oc apply -f -
-      log "✅ Red Hat registry pull secret created with ownerReference"
-    else
-      # Fallback: create without ownerReference
-      log "⚠️  TEKTON_PIPELINERUN_NAME/UID not set - creating secret without ownerReference"
-      log "⚠️  Manual cleanup required after test completion"
-
-      oc create secret docker-registry redhat-registry-pull-secret \
-        --docker-server=registry.redhat.io \
-        --docker-username="$REDHAT_USERNAME" \
-        --docker-password="$REDHAT_PASSWORD" \
-        -n "$NAMESPACE" 2>/dev/null && log "✅ Red Hat registry pull secret created" || log "⚠️  Secret exists or creation failed"
-    fi
-
-    # Link to default service account
+    oc create secret docker-registry redhat-registry-pull-secret \
+      --docker-server=registry.redhat.io \
+      --docker-username="$REDHAT_USERNAME" \
+      --docker-password="$REDHAT_PASSWORD" \
+      -n "$NAMESPACE" \
+      --dry-run=client -o yaml | oc apply -f -
+    log "✅ Red Hat registry pull secret created"
     oc secrets link default redhat-registry-pull-secret --for=pull -n "$NAMESPACE" 2>/dev/null || echo "⚠️  Secret already linked to default SA"
   else
     log "⚠️  Red Hat registry credentials not found in /var/run - OKP image pull may fail"
@@ -425,10 +361,8 @@ fi
 # Debug hook/port churn: export E2E_OPS_VERBOSE=1 before running pipeline.sh
 export E2E_LSC_PORT_FORWARD_PID_FILE="${E2E_LSC_PORT_FORWARD_PID_FILE:-/tmp/e2e-lightspeed-port-forward.pid}"
 export E2E_LLAMA_PORT_FORWARD_PID_FILE="${E2E_LLAMA_PORT_FORWARD_PID_FILE:-/tmp/e2e-llama-port-forward.pid}"
-export E2E_OKP_PORT_FORWARD_PID_FILE="${E2E_OKP_PORT_FORWARD_PID_FILE:-/tmp/e2e-okp-port-forward.pid}"
 rm -f "$E2E_LSC_PORT_FORWARD_PID_FILE"
 rm -f "$E2E_LLAMA_PORT_FORWARD_PID_FILE"
-rm -f "$E2E_OKP_PORT_FORWARD_PID_FILE"
 
 oc label pod lightspeed-stack-service pod=lightspeed-stack-service -n $NAMESPACE
 
@@ -449,7 +383,7 @@ kill_listeners_on_ports() {
     fi
   done
 }
-kill_listeners_on_ports 8080 8000 8321 8081
+kill_listeners_on_ports 8080 8000 8321
 
 # Start port-forward for lightspeed-stack
 progress "Starting port-forward, then E2E tests"
@@ -469,9 +403,6 @@ oc port-forward svc/llama-stack-service-svc 8321:8321 -n $NAMESPACE &
 PF_LLAMA_PID=$!
 echo "$PF_LLAMA_PID" >"$E2E_LLAMA_PORT_FORWARD_PID_FILE"
 
-# OKP Solr port-forward (localhost:8081) is started by e2e-ops deploy-okp-solr
-# when okp_rag.feature runs — do not start it here.
-
 # Wait for port-forward to be usable (app may not be listening immediately; port-forward can drop)
 log "Waiting for port-forward to lightspeed-stack to be ready..."
 for i in $(seq 1 36); do
@@ -581,13 +512,6 @@ if [[ -n "${E2E_LLAMA_PORT_FORWARD_PID_FILE:-}" && -f "$E2E_LLAMA_PORT_FORWARD_P
   fi
   rm -f "$E2E_LLAMA_PORT_FORWARD_PID_FILE"
 fi
-if [[ -n "${E2E_OKP_PORT_FORWARD_PID_FILE:-}" && -f "$E2E_OKP_PORT_FORWARD_PID_FILE" ]]; then
-  read -r _okp_pf <"$E2E_OKP_PORT_FORWARD_PID_FILE" 2>/dev/null || true
-  if [[ "${_okp_pf:-}" =~ ^[0-9]+$ ]]; then
-    kill -9 "$_okp_pf" 2>/dev/null || true
-  fi
-  rm -f "$E2E_OKP_PORT_FORWARD_PID_FILE"
-fi
 
 kill $PF_LCS_PID 2>/dev/null || true
 kill $PF_JWKS_PID 2>/dev/null || true