ci(railway): retry service resolution to survive the bootstrap→configure race by mmabrouk · Pull Request #5322 · Agenta-AI/agenta

mmabrouk · 2026-07-14T20:54:02Z

Context

The v0.105.0 release PR's preview deploy (run 29363754807) failed with Service 'cron' not found about 40 seconds into the deploy job. The setup job's bootstrap had created the cron service 17 seconds earlier and reported success. configure.sh resolves service ids from a single railway status --json snapshot taken at startup, and a service created seconds before by a different job can be missing from that snapshot (Railway's API is eventually consistent). The CLI fallback (railway variable set --service cron) hit the same window, and railway_call classifies its "not found" as a deterministic error, so nothing retried and the deploy hard-failed. The dependent eu/acceptance web tests were skipped with it. The same job flips between green and red on unchanged branches, which is how this was diagnosed as a race rather than a regression.

Changes

Two bounded retries in configure.sh, both no-ops on the happy path:

_service_id_with_retry: when a service id is missing from the cached status snapshot, re-fetch railway status --json and retry resolution (6 attempts total, 5s sleeps; tunable via RAILWAY_SERVICE_RESOLVE_ATTEMPTS / RAILWAY_SERVICE_RESOLVE_DELAY) before falling back to the CLI path. A first-try hit costs zero extra API calls and zero sleeps. The default is 6 because in the observed incident the service was still invisible more than 35 seconds after bootstrap completed, so the retry budget (25s of resolve retries, roughly 35s including the CLI fallback) is sized to cover it; unneeded retries cost nothing.
_cli_set_vars: the CLI fallback now retries specifically the "not found" failure (3 attempts, RAILWAY_CLI_SET_ATTEMPTS) instead of failing the deploy on the first hit. Any other failure surfaces immediately; transient network errors keep being handled inside railway_call as before.

Before: one status snapshot, one CLI attempt, Service 'cron' not found, deploy dead.
After: the snapshot is re-fetched while the just-created service becomes visible; the deploy only fails after both paths exhaust their bounded retries.

Tests / notes

I cannot test against live Railway from this environment. Happy-path behavior is unchanged by construction: first-try resolution takes the same code path with zero extra calls.
bash -n passes. shellcheck was not available locally.
Verified the retry logic with a local stubbed-railway_call harness (not committed): first-try hit makes no extra calls; an id appearing after a snapshot re-fetch resolves; exhaustion returns empty and falls back to the CLI; the CLI fallback retries "not found" and succeeds on the second attempt; deterministic errors (unauthorized) are not retried; a permanent "not found" still fails after the bounded 3 attempts. All 10 assertions pass.
To validate for real: rerun the preview deploy ("14 - check PR preview") on this PR and on the release PR after merge; the deploy job checks out this configure.sh.
Known sibling flake, not fixed here: deploy-gateway.sh failed on another PR with Project "agenta-oss-pr-5319" not found in workspace at railway link. That is a different script and failure point (raw railway link, project-level visibility), not the same one-line retry shape, so it is left for a follow-up if it recurs.

https://claude.ai/code/session_01Hyn9365BLPXDmNZShrQkmH

vercel · 2026-07-14T20:54:09Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jul 14, 2026 9:00pm

mmabrouk · 2026-07-14T20:54:10Z

@coderabbitai review

coderabbitai · 2026-07-14T20:54:12Z

📝 Walkthrough

Walkthrough

Railway service-variable configuration now refreshes status data, retries service discovery and CLI updates, and falls back to CLI variable setting when GraphQL service resolution remains unavailable.

Changes

Railway variable configuration resilience

Layer / File(s)	Summary
Status refresh and retry helpers `hosting/railway/oss/scripts/configure.sh`	Adds helpers to preserve refreshed Railway status data, retry service ID resolution with configurable attempts and delays, and retry CLI variable updates when services are not yet visible.
Variable upsert fallback `hosting/railway/oss/scripts/configure.sh`	Updates `upsert_service_vars` to use retrying service resolution and fall back to CLI variable setting when no service ID is resolved.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: adding retries to survive Railway service-resolution races during bootstrap/configure.
Description check	✅ Passed	The description is detailed and directly describes the same Railway retry and fallback changes in the pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix-railway-configure-race

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai · 2026-07-14T20:54:16Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…gure race The release preview deploy failed with "Service 'cron' not found" seconds after bootstrap created the service: configure.sh resolves service ids from a single 'railway status --json' snapshot, and a service created moments earlier by the setup job may not be visible in it yet. The CLI fallback then hit the same eventual-consistency window and hard-failed the deploy. Re-fetch the status snapshot and retry id resolution a few times (bounded, short sleeps) before falling back to the CLI, and let the CLI fallback retry the specific 'not found' failure a couple of times instead of exiting on the first hit. First-try hits cost zero extra API calls, so the happy path is unchanged. Claude-Session: https://claude.ai/code/session_01Hyn9365BLPXDmNZShrQkmH

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (1)

hosting/railway/oss/scripts/configure.sh (1)
134-135: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider using a distinct delay variable for CLI retries.

_cli_set_vars reuses RAILWAY_SERVICE_RESOLVE_DELAY. To maintain configuration consistency with RAILWAY_CLI_SET_ATTEMPTS, consider providing a dedicated delay variable.
♻️ Proposed refactor
     local attempts="${RAILWAY_CLI_SET_ATTEMPTS:-3}"
-    local delay="${RAILWAY_SERVICE_RESOLVE_DELAY:-5}"
+    local delay="${RAILWAY_CLI_SET_DELAY:-5}"

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 012c6cf9-61f4-48cb-87b6-8073ffb706cb

📥 Commits

Reviewing files that changed from the base of the PR and between 14895df and b670d1d.

📒 Files selected for processing (1)

hosting/railway/oss/scripts/configure.sh

coderabbitai · 2026-07-14T20:57:47Z

+    done
+}


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Return failure if the retry loop is skipped.

If RAILWAY_CLI_SET_ATTEMPTS evaluates to 0 or an invalid non-numeric string, the loop won't execute, and the function will fall through to implicitly return 0 (success) without doing any work. Adding return 1 ensures a failure is properly reported in this edge case.

🛡️ Proposed fix

done + return 1 }

coderabbitai · 2026-07-14T20:57:47Z

+        _cli_set_vars "$service" "$@"
        return 0


🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Propagate the exit code of the CLI fallback.

Both CLI fallback paths explicitly return 0 after calling _cli_set_vars. If the CLI variable update fails, its non-zero exit code is masked, causing the function to incorrectly report success. This can lead to silent deployment failures with missing configuration variables.

hosting/railway/oss/scripts/configure.sh#L177-L178: Replace return 0 with return $? to correctly propagate the exit status.

hosting/railway/oss/scripts/configure.sh#L188-L189: Replace return 0 with return $? to correctly propagate the exit status.

📍 Affects 1 file

hosting/railway/oss/scripts/configure.sh#L177-L178 (this comment)

hosting/railway/oss/scripts/configure.sh#L188-L189

github-actions · 2026-07-14T21:01:18Z

Railway Preview Environment


Status	Destroyed (PR closed)

Updated at 2026-07-14T21:03:48.248Z

dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Jul 14, 2026

dosubot Bot added the ci/cd label Jul 14, 2026

mmabrouk force-pushed the fix-railway-configure-race branch from b670d1d to b46eb2a Compare July 14, 2026 20:56

coderabbitai Bot reviewed Jul 14, 2026

View reviewed changes

vercel Bot deployed to Preview July 14, 2026 21:00 View deployment

mmabrouk merged commit 2853e37 into release/v0.105.0 Jul 14, 2026
32 of 36 checks passed

This was referenced Jul 30, 2026

[fix] Repair incomplete Railway previews [AGE-4023] #5576

Closed

[5566] fix(ci): Fail the Railway preview bootstrap when a service is missing #5588

Merged

mmabrouk mentioned this pull request Aug 2, 2026

(chore) Railway previews: clone a template environment per PR instead of creating 13 services imperatively #5650

Closed

5 tasks

Navigation Menu

Uh oh!

ci(railway): retry service resolution to survive the bootstrap→configure race - #5322

ci(railway): retry service resolution to survive the bootstrap→configure race#5322
mmabrouk merged 1 commit into
release/v0.105.0from
fix-railway-configure-race

mmabrouk commented Jul 14, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

mmabrouk commented Jul 14, 2026

Uh oh!

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 14, 2026

Uh oh!

coderabbitai Bot Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mmabrouk commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Changes

Tests / notes

Uh oh!

vercel Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmabrouk commented Jul 14, 2026

Uh oh!

coderabbitai Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Railway Preview Environment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mmabrouk commented Jul 14, 2026 •

edited

Loading

vercel Bot commented Jul 14, 2026 •

edited

Loading

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading