[5650] chore(ci): Railway preview clone spike + CI auth check - #5658
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdded Railway preview spike research, findings, and runbook documents. Updated Railway volume and deployment readiness handling. Added shared agent-environment detection and conditional cleanup and skipping to two Playwright acceptance tests. ChangesRailway preview clone spike
Agent acceptance gating
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
d2d5de4 to
db22085
Compare
db22085 to
b694a2f
Compare
|
Spike code removed per review. This PR is now docs and evidence only: the research ( |
dosubot
Bot
added
size:XL
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d325106a-1026-4d5f-9e55-9bd90dbcd5a7
⛔ Files ignored due to path filters (1)
docs/design/railway-preview-clone-spike/spike/results/cycle-log.csvis excluded by!**/*.csv
📒 Files selected for processing (4)
docs/design/railway-preview-clone-spike/findings.mddocs/design/railway-preview-clone-spike/research/failure-history.mddocs/design/railway-preview-clone-spike/research/railway-capabilities.mddocs/design/railway-preview-clone-spike/runbook.md
| Mutations, per-service deploy mode: 1 `environmentCreate` + 8 | ||
| `serviceInstanceUpdate` + 13 `serviceInstanceDeployV2` + <=1 | ||
| `serviceDomainCreate` + 1 `environmentDelete` = **24**. Plus reads: 1-2 id | ||
| resolutions (amortized across cycles), 1 populate check, ~1 read per 15s of | ||
| deploy wait. Realistic total: **~35-60/cycle**, dominated by status polling. | ||
|
|
||
| Mutations, triggers mode: 1 create + 8 patches + 1 `environmentTriggersDeploy` | ||
| + <=1 domain + 1 delete = **12**, meeting the <=15 criterion if polling reads | ||
| are excluded (record both numbers in the spike report; reads and writes share | ||
| the hourly budget). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="docs/design/railway-preview-clone-spike/findings.md"
echo "== file exists =="
test -f "$file" && echo yes || echo no
echo "== line count =="
wc -l "$file"
echo "== relevant section 70-125 =="
sed -n '70,125p' "$file" | nl -ba -v70
echo "== searches for triggers/patch/mutation/ledger/final/run =="
rg -n -i "trigger[s]?|patch|mutation[s]?|ledger|final|run|calls?|15|8|16|12|deploy mode" "$file"Repository: Agenta-AI/agenta
Length of output: 320
🏁 Script executed:
#!/bin/bash
set -eu o pipefail
file="docs/design/railway-preview-clone-spike/findings.md"
echo "== file exists =="
test -f "$file" && echo yes || echo no
echo "== line count =="
wc -l "$file"
echo "== relevant section 70-125 =="
sed -n '70,125p' "$file" | nl -ba -v70
echo "== searches for triggers/patch/mutation/ledger/final/run =="
rg -n -i "trigger[s]?|patch|mutation[s]?|ledger|final|run|calls?|15|8|16|12|deploy mode" "$file"Repository: Agenta-AI/agenta
Length of output: 320
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="docs/design/railway-preview-clone-spike/findings.md"
echo "== relevant section 85-130 =="
sed -n '85,130p' "$file" | awk '{printf "%5d %s\n", NR+84, $0}'
echo "== all matching calls/modes/counters =="
python3 - <<'PY'
import re
from pathlib import Path
p = Path("docs/design/railway-preview-clone-spike/findings.md")
text = p.read_text()
pat = re.compile(r'(?i)\b(trigger[s]?\b|patch mode|deploy mode|mutation|calls|run|12|15|16|8|environmentTriggersDeploy|serviceInstanceDeployV2)')
for i, line in enumerate(text.splitlines(), 1):
if pat.search(line):
print(f"{i:03d}: {line}")
PYRepository: Agenta-AI/agenta
Length of output: 12559
Archive or update the static triggers-mode ledger.
Lines 89-98 still present environmentTriggersDeploy as a valid 12-call path, but the live findings disprove triggers mode and report the production-shaped patch-mode path as 8 mutations and 15 total API calls. Mark that ledger as historical or replace it with the final measurements.
| - **Visibility flip still pending — GitHub-UI-only.** New GHCR packages | ||
| default to PRIVATE and there is NO REST or GraphQL API to change package | ||
| visibility (`PATCH orgs/.../packages/...` returns 404; the docs describe | ||
| only the web UI "Danger Zone -> Change visibility" flow). Anonymous | ||
| manifest pulls return 401/403 while private, so Railway cannot pull them | ||
| yet. Once an org admin flips the three packages public, | ||
| `spike/switch-template-to-ghcr-images.sh` (pull-gated: it refuses to touch | ||
| the template while any package is private) switches the template to the | ||
| images and clears the startCommand overrides; run one patch-mode cycle to | ||
| certify the registry-backed path. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Synchronize all close-out status sections.
The same GHCR work is marked pending in multiple sections but complete in the findings addendum. Readers can follow obsolete commands or misread completed validation.
docs/design/railway-preview-clone-spike/findings.md#L205-L214: mark pending visibility as historical or update it.docs/design/railway-preview-clone-spike/findings.md#L306-L317: remove completed items from the remaining-work list.docs/design/railway-preview-clone-spike/runbook.md#L91-L108: update the status and remove the obsolete switch procedure.
📍 Affects 2 files
docs/design/railway-preview-clone-spike/findings.md#L205-L214(this comment)docs/design/railway-preview-clone-spike/findings.md#L306-L317docs/design/railway-preview-clone-spike/runbook.md#L91-L108
| - **Patch mode, `--image-tag pr-5651-a46168f`: 10 consecutive green rows** | ||
| (16:53-17:23 UTC), 55-72s per cycle, 16 total API calls per cycle (8 of | ||
| them mutations). Honesty note: between the 8th and 9th green row one cycle | ||
| attempt was ABORTED mid-run (no CSV row): the hourly rate budget — spent by | ||
| the day's debugging, not by the cycles — ran out and the operator stopped | ||
| the run rather than let 429-retry exhaustion record a bogus failure; its | ||
| clone was deleted, and the run resumed after the 17:22 UTC window reset. | ||
| No cycle ever failed for a design reason in either series. | ||
|
|
||
| | # | Criterion (issue #5650) | Verdict | | ||
| |---|--------------------------|---------| | ||
| | 1 | 10 consecutive green clone→patch→deploy→smoke→delete cycles | **MET** — per-service 10/10 strictly unbroken; patch mode 10 green rows with the aborted-attempt note above. | | ||
| | 2 | ≤ 15 Railway API calls per cycle | **MET on mutations, 1 over on total calls.** Patch mode: 8 mutations (create + 3 infra deploys + 1 patchCommit + supertokens + gateway + delete). Total HTTP requests incl. polling = 16: the 8 mutations + 8 reads (populate check, domain check, ~5 status polls at 15s, post-patch refresh). Merging the populate and domain reads or widening `RW_POLL_INTERVAL` brings the total to ≤15; mutations are the scarce/priced dimension and sit at nearly half the target. | | ||
| | 3 | Wall time ≤ today's setup+deploy | **MET with ~3x headroom.** Baseline measured live from the last successful `14-check-pr-preview.yml` deploy (run 30753555499, 2026-08-02): setup 38s + deploy 2m18s = **~176s**, excluding image builds. Patch-mode cycles complete clone→patch→deploy→smoke in ~51-55s (55-57s including the delete). | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the design findings file and inspect the relevant context plus nearby criteria/terms.
if [ -f docs/design/railway-preview-clone-spike/findings.md ]; then
echo "== file exists =="
wc -l docs/design/railway-preview-clone-spike/findings.md
echo
echo "== relevant section =="
sed -n '240,310p' docs/design/railway-preview-clone-spike/findings.md
echo
echo "== nearby criterion references =="
rg -n "10 consecutive|consecutive|rate-limit|ABORTED|abort|Metric|Criterion|issue `#5650`|patch mode|recorded green|interrup|unbroken" docs/design/railway-preview-clone-spike/findings.md || true
else
echo "missing docs/design/railway-preview-clone-spike/findings.md"
fiRepository: Agenta-AI/agenta
Length of output: 6310
Define whether aborted attempts invalidate the 10-consecutive-cycle criterion.
Patch mode reports 10 green CSV rows, but an ABORTED cycle attempt occurred between green rows 8 and 9. If the criterion requires 10 consecutive cycle attempts, run an uninterrupted set. If rate-limit-aborted attempts are excluded, change the criterion and verdict wording to make that explicit.
| - Operations the CLI lacks or does worse: `environmentCreate` with source, | ||
| `serviceCreate` with image source and inline variables, `serviceInstanceUpdate`, | ||
| `serviceInstanceDeployV2`, `volumeCreate`, `templateDeployV2`, bulk | ||
| `variableCollectionUpsert`. Nothing is documented as idempotent; idempotency must | ||
| be built client-side (query by name first). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Tracked files matching docs/design/railway-preview-clone-spike:"
git ls-files 'docs/design/railway-preview-clone-spike/*' | sort
echo
echo "Relevant sections:"
for f in \
docs/design/railway-preview-clone-spike/research/railway-capabilities.md \
docs/design/railway-preview-clone-spike/findings.md \
docs/design/railway-preview-clone-spike/runbook.md
do
if [ -f "$f" ]; then
echo "--- $f ($(wc -l < "$f") lines) ---"
sed -n '1,220p' "$f"
fi
done
echo
echo "Graph references to rw_find_project_id / projects / me and workspace:"
rg -n "rw_find_project_id|find_project|projects\((?!\))|me\s*\{|workspaces|duplicate names|first match|idempotent|idempotency|id \b|projectId|project id" docs/design/railway-preview-clone-spike -SRepository: Agenta-AI/agenta
Length of output: 33757
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Lib GraphQL project/workspace lookup:"
for f in \
docs/design/railway-preview-clone-spike/spike/lib-graphql.sh \
docs/design/railway-preview-clone-spike/spike/build-template-env.sh \
docs/design/railway-preview-clone-spike/spike/preview-cycle.sh
do
if [ -f "$f" ]; then
echo "--- $f ($(wc -l < "$f") lines) ---"
rg -n --pcre2 "rw_find_project_id|rw_.*project|projects|workspaces|me \\{|id|projectId|workspaceId|first match|duplicate|environmentCreate|environments" "$f" -C 3 || true
fi
done
echo
echo "Occurrences of project lookup helpers:"
rg -n --pcre2 "rw_find_project_id|function rw_find|projects\\(|workspaceId|first match|query by name|poll-by-name|environments\\(projectId" docs/design -S || trueRepository: Agenta-AI/agenta
Length of output: 2549
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Full lib-graphql.sh:"
wc -l docs/design/railway-preview-clone-spike/spike/lib-graphql.sh
cat -n docs/design/railway-preview-clone-spike/spike/lib-graphql.sh
echo
echo "Full runbook project_id usage:"
sed -n '200,240p' docs/design/railway-preview-clone-spike/runbook.md | cat -n
echo
echo "All Railway spike scripts:"
git ls-files 'docs/design/railway-preview-clone-spike/spike/*' | sort
for f in $(git ls-files 'docs/design/railway-preview-clone-spike/spike/*' | sort); do
echo "--- $f ($(wc -l < "$f") lines) ---"
rg -n --pcre2 "rw_find|project|workspace|project_id|environmentCreate|environment|me|projects|first|names|name" "$f" -C 2 || true
doneRepository: Agenta-AI/agenta
Length of output: 267
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Tracked files under docs/design/railway-preview-clone-spike:"
git ls-files 'docs/design/railway-preview-clone-spike' | sort
echo
echo "Current occurrences of lookup helpers / project variables:"
rg -n --pcre2 "rw_find|find_project|project_id|PROJECTS|workspaceId|first match|same name|duplicate|query by name|poll-by-name|environmentCreate|environments\\s*\\(projectId|environment\\(id\\)|environment\\(input|workspace\\(id\\)" docs/design/railway-preview-clone-spike docs/project -S 2>/dev/null || true
echo
echo "Current runbook project_id usage:"
if [ -f docs/design/railway-preview-clone-spike/runbook.md ]; then
sed -n '180,245p' docs/design/railway-preview-clone-spike/runbook.md | cat -n
fiRepository: Agenta-AI/agenta
Length of output: 6303
Use exact IDs for Railway resource reconciliation.
Name lookup is not idempotent, and the project lookup finding still describes first-match for duplicate preview project names. Document exact-ID reconciliation and require a unique project/environment match, and remove the first-match wording from findings.md#L132-L136.
📍 Affects 2 files
docs/design/railway-preview-clone-spike/research/railway-capabilities.md#L63-L67(this comment)docs/design/railway-preview-clone-spike/findings.md#L132-L136
| Exact steps to run the spike end to end. Everything below runs from the repo | ||
| root. All scripts live in `docs/design/railway-preview-clone-spike/spike/`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Mark the procedure as historical or update every command to a live path.
The preamble says the spike scripts were removed, but this runbook still instructs users to source and execute files under docs/design/railway-preview-clone-spike/spike/. Replace these commands with the productionized paths named in Lines 3-8, or label the sections as archived evidence and remove “Exact steps to run.”
Also applies to: 60-63, 76-108, 115-155, 169-189, 204-219, 226-237
| The template is intentionally left **undeployed** — whether an undeployed | ||
| template clones into a deployable environment is spike question Q5 | ||
| (`findings.md`). To debug the template itself, re-run with `--deploy`, which | ||
| triggers all 13 deploys and smoke-tests the template env. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate relevant files"
git ls-files | rg '(^|/)docs/design/railway-preview-clone-spike/(runbook|findings)\.md$|AGENTS\.md$' || true
echo
echo "runbook excerpt around lines 140-180"
if [ -f docs/design/railway-preview-clone-spike/runbook.md ]; then
sed -n '140,180p' docs/design/railway-preview-clone-spike/runbook.md | nl -ba -v140
fi
echo
echo "findings references to deploy / 13 / undeployed / triggered"
if [ -f docs/design/railway-preview-clone-spike/findings.md ]; then
rg -n -i -C 3 '(deploy|deployed|13|undeployed|clones|template|service|image)' docs/design/railway-preview-clone-spike/findings.md || true
fi
echo
echo "Search runbook for --deploy / undeployed / Q5"
if [ -f docs/design/railway-preview-clone-spike/runbook.md ]; then
rg -n -C 3 '(--deploy|undeployed|Q5|findings\.md|13 deploys|deployments|deployed)' docs/design/railway-preview-clone-spike/runbook.md || true
fiRepository: Agenta-AI/agenta
Length of output: 570
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "runbook excerpt around lines 140-180"
if [ -f docs/design/railway-preview-clone-spike/runbook.md ]; then
sed -n '140,180p' docs/design/railway-preview-clone-spike/runbook.md | awk '{printf "%6s\t%s\n", NR+139, $0}'
fi
echo
echo "findings references to deploy / 13 / undeployed / triggered"
if [ -f docs/design/railway-preview-clone-spike/findings.md ]; then
rg -n -i -C 3 '(deploy|deployed|13|undeployed|clones|template|service|image)' docs/design/railway-preview-clone-spike/findings.md || true
fi
echo
echo "Search runbook for --deploy / undeployed / Q5"
if [ -f docs/design/railway-preview-clone-spike/runbook.md ]; then
rg -n -C 3 '(--deploy|undeployed|Q5|findings\.md|13 deploys|deployments|deployed)' docs/design/railway-preview-clone-spike/runbook.md || true
fi
echo
echo "Findings file size"
wc -l docs/design/railway-preview-clone-spike/findings.md docs/design/railway-preview-clone-spike/runbook.md 2>/dev/null || trueRepository: Agenta-AI/agenta
Length of output: 29597
Drop the undeployed-template claim.
railway add --service --image creates deployments for all 13 template services, so this command does not leave the template undeployed. If --deploy is still intended for debugging after deploy-mode fixes, describe a command path that keeps initial deployments off, or state the cost of accepting deployed template services.
|
One functional commit added on top of the docs: the #5671 fix ( |
Railway Preview Environment
Updated at 2026-08-03T06:54:24.162Z |
|
Second small fix commit added (test hygiene, ref #5673): the agent-platform Playwright specs now skip with a clear reason when the environment cannot create agent-type apps (flag-off OSS previews silently yield prompt apps), and archive the misclassified seed so it cannot pollute other specs' app lookups. This is what makes the preview web suite meaningful again repo-wide; page-snapshot evidence is on #5673. |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
web/oss/tests/playwright/acceptance/utils/agentApps.ts (1)
19-24: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winGuard the
about:blankfallback and reuse this helper instead of duplicating it.
page.url()returns"about:blank"on a page that has not navigated yet. That value is truthy, sopage.url() || process.env.AGENTA_WEB_URL || "http://localhost:3000"never falls through to the intended fallback.new URL("about:blank").originthen evaluates to the literal string"null", producing an apiBase of"null/api"for any caller that invokes this before the page navigates. Sincepage.request.postcalls made against that base will fail (caught by.catch(() => null)inqueryWorkflowAgentState), the failure surfaces only as a 15-second timeout ending in"unknown", not as a clear error.Additionally,
agent-chat/tests.tsstill keeps its own separateapiBasefunction with identical logic instead of importing this new export. Consolidate on this helper to avoid two copies of the same bug.🐛 Proposed fix
export const resolveApiBase = (page: Page): string => { if (process.env.AGENTA_API_URL) return process.env.AGENTA_API_URL - const origin = new URL(page.url() || process.env.AGENTA_WEB_URL || "http://localhost:3000") - .origin + const pageUrl = page.url() + const origin = new URL( + pageUrl && pageUrl !== "about:blank" + ? pageUrl + : process.env.AGENTA_WEB_URL || "http://localhost:3000", + ).origin return `${origin}/api` }Please confirm that
page.url()returns"about:blank"for un-navigated pages in the Playwright version pinned for this repo, to validate the fix.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: aacebcf9-97d9-4f78-b8ed-9c9928c9a56f
📒 Files selected for processing (3)
web/oss/tests/playwright/acceptance/agent-chat/tests.tsweb/oss/tests/playwright/acceptance/agent-skills/skill-folder-upload.spec.tsweb/oss/tests/playwright/acceptance/utils/agentApps.ts
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
hosting/railway/oss/scripts/deploy-from-images.sh (1)
79-80: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument the readiness polling controls.
Add
RAILWAY_SERVICE_ACTIVE_ATTEMPTSandRAILWAY_SERVICE_ACTIVE_DELAYtohosting/railway/oss/README.mdLines 118-127, including their defaults and effective timeout.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 669d7651-0a73-4de7-bafb-e09433f233b6
📒 Files selected for processing (1)
hosting/railway/oss/scripts/deploy-from-images.sh
| for ((attempt = 1; attempt <= attempts; attempt++)); do | ||
| status="$(railway_call status --json 2>/dev/null \ | ||
| | jq -r --arg env "$ENV_NAME" --arg svc "$service" \ | ||
| '.environments.edges[].node | select(.name == $env) | ||
| | .serviceInstances.edges[].node | select(.serviceName == $svc) | ||
| | .latestDeployment.status // "NONE"' 2>/dev/null | head -n 1)" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate file =="
fd -a 'deploy-from-images\.sh$' . || true
echo "== File outline / size =="
wc -l hosting/railway/oss/scripts/deploy-from-images.sh
ast-grep outline hosting/railway/oss/scripts/deploy-from-images.sh || true
echo "== Relevant sections =="
sed -n '1,150p' hosting/railway/oss/scripts/deploy-from-images.sh
echo "===== 250-300 ====="
sed -n '250,305p' hosting/railway/oss/scripts/deploy-from-images.sh
echo "== Railway CLI status/redeploy usages =="
rg -n "railway_call|latestDeployment|redeploy|create deployment|deployment" hosting/railway/ -SRepository: Agenta-AI/agenta
Length of output: 20226
🌐 Web query:
Railway API manage deployments status query deployment id latest deployment SUCCESS documentation
💡 Result:
To manage and query deployment statuses on Railway, you use the public GraphQL API [1][2]. Railway deployments progress through several states, where SUCCESS indicates the deployment is running successfully [1][3]. GraphQL Query to Check Deployment Status To check the status of a specific deployment or the latest deployments for a service, use the deployments query [4][5]. Example query for the latest deployment status of a service: query GetLatestDeploymentStatus($serviceId: String!, $environmentId: String!) { deployments(first: 1, input: {serviceId: $serviceId, environmentId: $environmentId}) { edges { node { id status } } } } In this query, the status field returns the current state of the deployment [4]. Common statuses include BUILDING, DEPLOYING, SUCCESS, FAILED, CRASHED, and QUEUED [1][3]. Managing Deployments via API - Triggering: You can trigger a new deployment using the serviceInstanceDeployV2 mutation [6][7]. - Redeploying: Use the deploymentRedeploy or serviceInstanceRedeploy mutations to trigger a redeploy of existing code [6][7]. - Monitoring: Because Railway's CLI and API operations for triggering deployments are often asynchronous (they trigger the action and exit), a common pattern for CI/CD pipelines is to poll the GraphQL API for the status until it reaches a terminal state like SUCCESS or FAILED [5]. Official Documentation For comprehensive details on all available GraphQL operations, refer to the official Railway API documentation at https://docs.railway.com/integrations/api/manage-deployments [1] and the Railway GraphQL playground (https://railway.com/graphiql) [8][9].
Citations:
- 1: https://docs.railway.com/integrations/api/manage-deployments
- 2: https://docs.railway.com/integrations/api
- 3: https://docs.railway.com/cli/deployment
- 4: https://station.railway.com/questions/get-all-deployments-with-graph-ql-api-f416e08c
- 5: https://station.railway.com/questions/deploying-pre-built-images-from-git-hub-a-d4ac84bd
- 6: https://docs.railway.com/integrations/api/manage-services
- 7: https://apis.io/apis/railway-app/railway-app-deployments-api/
- 8: https://github.com/railwayapp/docs/blob/76db91a4/content/docs/deployments/reference.md
- 9: https://docs.railway.com/deployments/reference
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== railway_call and lock implementations =="
sed -n '100,180p' hosting/railway/oss/scripts/lib.sh
sed -n '280,350p' hosting/railway/oss/scripts/lib.sh
echo "== All railway_call status path parsing =="
rg -n "railway_call status --json|latestDeployment|serviceInstanceDeploy|deploymentRedeploy|id:" hosting/railway -S
echo "== Check railway CLI status output in installed CLI docs/help if present =="
type railway >/dev/null 2>&1 && {
railway version >/dev/null 2>&1;
railway help redeploy 2>/dev/null || true;
railway help status 2>/dev/null || true;
} || true
echo "== Static verifier: redeploy and wait helpers are separate functions without deployment id =="
python3 - <<'PY'
from pathlib import Path
p = Path("hosting/railway/oss/scripts/deploy-from-images.sh")
text = p.read_text()
checks = {
"redeploy discards stdout": 'err="$(railway_call redeploy --service "$service" --environment "$ENV_NAME" --yes 2>&1 >/dev/null)"' in text,
"wait receives only service": 'wait_for_service_active() {
local service="$1"' in text,
"wait polls latestDeployment.status": '.latestDeployment.status // "NONE"' in text,
"wait accepts SUCCESS": 'SUCCESS)' in text[text.index("wait_for_service_active()"):text.index("run_alembic_with_retries()")],
"call sites do not pass deployment id": 'wait_for_service_active "$POSTGRES_SERVICE"' in text,
}
for name, ok in checks.items():
print(f"{name}: {ok}")
PYRepository: Agenta-AI/agenta
Length of output: 7539
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Static verifier: deployment identity ordering =="
python3 - <<'PY'
from pathlib import Path
p = Path("hosting/railway/oss/scripts/deploy-from-images.sh")
text = p.read_text()
for name, needle in [
("redeploy discards stdout", 'err="$(railway_call redeploy --service "$service" --environment "$ENV_NAME" --yes 2>&1 >/dev/null)"'),
("wait receives only service", 'wait_for_service_active() {\n local service="$1"'),
("wait polls latestDeployment.status", '.latestDeployment.status // "NONE"'),
("wait accepts SUCCESS", "SUCCESS)"),
("call site passes only service", 'wait_for_service_active "$POSTGRES_SERVICE"'),
("call site checks redeployed service only", 'redeploy_service_if_exists "$POSTGRES_SERVICE"\nwait_for_service_active "$POSTGRES_SERVICE"'),
]:
print(f"{name}: {needle in text}")
# Track lexical order of redeploy and wait; accept either because both functions can exist in any order.
start = text.index("redeploy_service_if_exists()")
end = text.index("wait_for_service_active()")
middle = text[start:end]
redeploy_return = "return $exit_code" in middle
wait_returns_status = "local attempt status" in text[text.index("wait_for_service_active()"):text.index("run_alembic_with_retries()")]
print(f"wait_for_service_active does not capture deployment id: {'deploymentId' not in text[text.index('wait_for_service_active()'):text.index('run_alembic_with_retries()')] and 'id' not in text[text.index('wait_for_service_active()'):text.index('run_alembic_with_retries()')]}")
PY
echo "== Railway docs/deploy script snippets for readiness pattern =="
if [ -f hosting/railway/oss/README.md ]; then
rg -n -C 3 "redeploy|DNS|Postgres|wait" hosting/railway/oss/README.md
fiRepository: Agenta-AI/agenta
Length of output: 4765
Bind the success wait to the redeployed deployment.
redeploy_service_if_exists starts a new deployment, but wait_for_service_active only passes the service name and accepts any SUCCESS value from .latestDeployment.status. If Railway reports a stale deployment before the new one is ready, Alembic can start against the old Postgres instance and fail. Capture the new deployment identity from the redeploy response, then poll that deployment until it is active.
| case "$status" in | ||
| SUCCESS) | ||
| return 0 | ||
| ;; | ||
| FAILED|CRASHED) | ||
| printf "Service '%s' deployment reached terminal status %s after redeploy.\n" \ | ||
| "$service" "$status" >&2 | ||
| return 1 | ||
| ;; | ||
| esac |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repository files of interest:\n'
git ls-files | rg 'hosting/railway/oss/scripts/deploy-from-images\.sh|AGENTS\.md|railway|deploy' | sed -n '1,120p'
printf '\nFile outline/text relevant lines:\n'
wc -l hosting/railway/oss/scripts/deploy-from-images.sh 2>/dev/null || true
sed -n '1,140p' hosting/railway/oss/scripts/deploy-from-images.sh 2>/dev/null || true
printf '\nRailway status references in repo:\n'
rg -n "status|REMOVED|REMOVING|SKIPPED|SLEEPING|FAILED|CRASHED|SUCCESS|timeout|sleep" hosting -S | sed -n '1,240p'Repository: Agenta-AI/agenta
Length of output: 31522
🌐 Web query:
Railway API manage deployments status REMOVED REMOVING SKIPPED SLEEPING
💡 Result:
In the Railway API, REMOVED, REMOVING, SKIPPED, and SLEEPING are valid states for a deployment [1][2]. These statuses are part of the broader deployment lifecycle, which can be queried via the Railway GraphQL API [3][1]. The specific meanings of these statuses are: REMOVING: This indicates that a deployment is currently in the process of being removed [3][4]. Older deployments (such as those previously in an Active, Completed, or Crashed state) transition to this status before being fully removed when a new deployment is triggered [3][4]. REMOVED: This is the final state for a deployment that has been deleted or cleaned up [3][4]. Deployments transition to this state after passing through the REMOVING phase [3][4]. Deployments may also be marked as REMOVED when they are stopped manually or aborted during the initialization/building phase [3]. SKIPPED: This indicates that a deployment was bypassed or not executed [5][1]. SLEEPING: This indicates that a deployment is inactive or "sleeping" [5][1]. Railway services may enter a sleeping state under certain conditions to manage resources [5]. These statuses are essential for tracking the full history of deployments within a project, including those that are no longer active [3][6]. When programmatically managing or monitoring deployments via the API or CLI, it is recommended to poll the deployment list and handle these states appropriately rather than assuming all non-SUCCESS deployments are errors [2][7].
Citations:
- 1: https://raw.githubusercontent.com/api-evangelist/railway-app/refs/heads/main/graphql/railway-app-schema.graphql
- 2: https://github.com/railwayapp/railway-skills/blob/main/plugins/railway/skills/use-railway/SKILL.md
- 3: https://docs.railway.com/deployments/reference
- 4: https://github.com/railwayapp/docs/blob/76db91a4/content/docs/deployments/reference.md
- 5: https://docs.railway.com/integrations/api/manage-deployments
- 6: https://github.com/railwayapp/cli/blob/a8a5afe/src/commands/deployment.rs
- 7: https://claudeskills.info/skills/railwayapp/railway-skills/use-railway/
Handle terminal non-running Railway deployment statuses immediately.
The current case only stops waiting for SUCCESS, FAILED, or CRASHED. Railway also reports terminal statuses such as REMOVED, SKIPPED, and potentially SLEEPING. Log the exact status and fail early instead of waiting the full 300s timeout for conditions that cannot become active.
1 participant
Draft, do not merge. This is the validated spike workspace for #5650 plus the one check that must run in CI to close the design's last open question.
What this proves
Railway PR previews currently issue ~55-75 non-idempotent CLI mutations per run to build 13 services from nothing, which is why they keep failing in new ways (22 patched failure modes cataloged in
research/failure-history.md). The spike validates the replacement: keep one fully configured template environment, and per PR do one clone call, one batched image-tag patch, deploy, smoke, delete.Live results against a throwaway Railway project (
spike/results/cycle-log.csv):environmentPatchCommitmode: 8 mutations / 16 total calls / ~55s per full clone-deploy-smoke-delete cycle, versus a 176s measured setup+deploy baseline today.latestmust never be the template tag), and upload-built services do not survive cloning, so gateway/redis/seaweedfs need self-contained images (spike/images/) or the validated start-command fallback (spike/apply-wrapper-startcommands.sh).What this PR runs in CI
46-railway-clone-auth-check.ymlexecutes on this PR (path-scoped) using the repo'sRAILWAY_TOKENsecret: an account-levelmequery, one template clone withskipInitialDeploys, and the clone's deletion, printing rate-limit headers. Green here answers the historical "auth and rights problems with Railway in Actions" concern for the exact operations the new design needs, including the two known traps (the secret must be exported asRAILWAY_API_TOKEN, and workspace tokens cannot create environments).Not in this PR
No production workflow changes. The migration itself (self-contained wrapper images in the build, template definition as code, swapping workflows 41/43/45) is the phased plan in #5650.
Refs #5650
https://claude.ai/code/session_011zdniFW44QDqcs4Nz8n3hr