Skip to content

Replace JSON timeline/A2A stores with shared SQL persistence (SQLite/Postgres) - #118

Draft
groupthinking with Copilot wants to merge 2 commits into
mainfrom
copilot/update-storage-approach-for-railway
Draft

Replace JSON timeline/A2A stores with shared SQL persistence (SQLite/Postgres)#118
groupthinking with Copilot wants to merge 2 commits into
mainfrom
copilot/update-storage-approach-for-railway

Conversation

Copilot AI commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The timeline and A2A flows depended on per-service JSON files, which breaks cross-service state sharing on Railway and is race-prone under concurrent writers. This change moves both stores to a shared SQL backend selected via DATABASE_URL, preserving existing store APIs used by timeline_server.py, listener.py, and mcp_dispatcher.py.

  • Storage backend + engine lifecycle

    • Added storage_db.py with SQLAlchemy-backed engine/schema for timeline_items, a2a_agents, and a2a_messages.
    • Backend selection via DATABASE_URL:
      • default/unset → SQLite (~/.xmcp/xmcp.db)
      • postgres://... normalized to postgresql://... for Railway compatibility.
    • Auto-creates schema at startup (create_all).
    • SQLite local multi-process safety: WAL mode + busy timeout.
  • Drop-in store refactor (API preserved)

    • Replaced file I/O in timeline_store.py and a2a_store.py with SQL operations while keeping function signatures unchanged.
    • Preserved payload shape (including nested fields) by storing actions, tags, and metadata as JSON columns.
  • Concurrency correctness

    • timeline_store.update_item() now executes read/merge/write in one transaction using a write lock at the DB transaction level (SQLite uses BEGIN IMMEDIATE), preventing lost updates in approval/action paths.
  • One-time migration from legacy JSON

    • Added scripts/migrate_json_to_sql.py to import TIMELINE_STORE_PATH and A2A_STORE_PATH JSON data.
    • Script is idempotent (upsert semantics by stable IDs), safe to run multiple times.
  • Config + deployment docs aligned with shared DB model

    • Added DATABASE_URL guidance to env.example and .env.example.
    • Updated docker-compose.yml to include Postgres and wire shared DATABASE_URL across services.
    • Updated README.md and docs/DEPLOYMENT.md to describe SQL storage and Railway requirement that all four services share one Postgres DATABASE_URL.
  • Coverage for new behavior

    • Added tests/test_sql_storage.py for:
      • timeline CRUD + A2A CRUD on SQLite
      • concurrent timeline updates (no lost update)
      • postgres:// normalization
      • JSON→SQL migration idempotency
# storage_db.py
def normalize_database_url(url: Optional[str]) -> str:
    raw = (url or "").strip()
    if not raw:
        return f"sqlite:///{DEFAULT_DB_PATH}"
    if raw.startswith("postgres://"):
        return "postgresql://" + raw[len("postgres://"):]
    return raw

Copilot AI requested review from Copilot and removed request for Copilot August 2, 2026 09:00
Copilot AI requested review from Copilot and removed request for Copilot August 2, 2026 09:06
Copilot AI changed the title [WIP] Update storage approach for Railway deployment Replace JSON timeline/A2A stores with shared SQL persistence (SQLite/Postgres) Aug 2, 2026
Copilot AI requested a review from groupthinking August 2, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants