diff --git a/docs/design/railway-preview-clone-spike/findings.md b/docs/design/railway-preview-clone-spike/findings.md new file mode 100644 index 0000000000..3796370db1 --- /dev/null +++ b/docs/design/railway-preview-clone-spike/findings.md @@ -0,0 +1,336 @@ +# Findings: Railway clone-based preview spike (issue #5650) + +Status date: 2026-08-02. Schema facts below were verified against the real API +via **unauthenticated introspection** (confirmed working: schema queries +against `https://backboard.railway.com/graphql/v2` succeed with no +`Authorization` header). Live-run rows are updated as phases execute. + +## Open questions from issue #5650 + +| # | Question | Status | Answered by | +|---|----------|--------|-------------| +| Q1 | `environmentCreate` 504 behavior at 13-service scale: does the clone complete in the background, and does poll-by-name recover it reliably? | **answered live (2026-08-02): no 504 across 25+ clones** — every `environmentCreate` returned synchronously in 3-4s (`clone_s` column across both 10-cycle series). The poll-by-name fallback stays as unexercised insurance. | `results/cycle-log.csv`, both 10-cycle runs. | +| Q2 | Clone fidelity: reference variables. Do `${{gateway.RAILWAY_PUBLIC_DOMAIN}}`, `${{Postgres.POSTGRES_PASSWORD}}` etc. stay references (re-resolving inside the clone) rather than freezing to template values? | **answered live (2026-08-02): yes, references re-resolve per clone.** Alembic connected to the CLONE's Postgres through `${{Postgres.*}}` references; the full smoke passed through the clone's own regenerated gateway domain. | Kept clone: migrations + 200 on all three smoke paths. | +| Q3 | Clone fidelity: volumes (recreated empty and attached?) and service domains (expected NOT copied). | **answered live (2026-08-02): both better than expected.** Volumes: recreated empty and attached — clone Postgres booted on its own volume and alembic created all three databases from scratch. Domains: Railway REGENERATED a service domain for the gateway in the clone automatically (`gateway-.up.railway.app`), so no `serviceDomainCreate` is needed (the script keeps it as a fallback). | Observed in the kept clone `pr-clone-c1-1785674907`. | +| Q4 | Clone fidelity: private networking. Does `*.railway.internal` resolve per-environment inside the clone? | **answered live (2026-08-02): yes.** `/w`, `/api/health`, `/services/health` all returned 200 through the clone's gateway (nginx -> web/api/services over `*.railway.internal`), and api reached Postgres/redis/supertokens internally. | Kept clone smoke, 2026-08-02. | +| Q5 | Can template services stay permanently undeployed while still cloning into deployable environments? | **answered live (2026-08-02): yes.** The 16:23+ cycles cloned from a template whose services showed `latestDeployment: NONE` (the earlier CLI-triggered deployments had been removed), and every clone deployed green. Template deployments are irrelevant to clones in both directions (stale ones don't leak, missing ones don't block). | Both 10-cycle series in `results/cycle-log.csv`. | +| Q6 | CI auth: does the shared `secrets.RAILWAY_TOKEN` (account token) work from GitHub Actions for `me`, clone, delete — and what rate-limit tier do the headers report? | pending live run | Runbook phase 4: `spike/ci-auth-check.yml`. | +| Q7 | Does `serviceInstanceUpdate` apply immediately, or does it only STAGE a change (staged-changes mutations exist in the schema: `environmentStageChanges`, `environmentPatchCommit*`, `environmentApplyChangeSet`) that a deploy would ignore? | **answered live (2026-08-02): applies immediately.** No `unmergedChangesCount` warning fired, and the clone deployment's `meta.serviceManifest` carried the patched `image`, the fixup `startCommand`, and the `ON_FAILURE`/10 restart policy verbatim. | Verified via `deployment(id).meta` of a clone deployment. | +| Q8 | API calls per cycle: can a cycle fit in <=15 calls? | **answered live (2026-08-02): yes on mutations (8), 16 on total HTTP calls incl. polling.** The low-call mechanism is `environmentPatchCommit` (NOT `environmentTriggersDeploy`, which is per-service — see the deploy-mode section). Full verdict in the Results section. | Patch-mode rows in `results/cycle-log.csv`; Results section below. | +| Q9 | Deployment status semantics for a one-shot service (alembic runs migrations and exits): does `latestDeployment.status` settle on `SUCCESS`, or `SLEEPING`/`REMOVED`? | **answered live (2026-08-02): stays `SUCCESS`.** With `restartPolicyType: ON_FAILURE` the exit-0 container is not restarted and the deployment keeps reporting SUCCESS. | Kept clone, alembic after migrations completed. | +| Q10 | Is `startCommand` executed through a shell (needed for the alembic inline `psql` command)? | **answered live (2026-08-02): NO.** Railway tokenizes the string (quote-aware) and execs it argv-style. A bare `until ...` start command fails within seconds as FAILED with EMPTY build and deploy logs (the misleading signature that cost cycle 1). Wrapping in `sh -c '