-
Notifications
You must be signed in to change notification settings - Fork 0
feat(wired) : wired everything to make the pipeline #7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,154 +1,246 @@ | ||
| # void-server | ||
|
|
||
| > **Turborepo Monorepo with Dockerized PostgreSQL + Redis, Node.js (Express), and Python (FastAPI)** | ||
| > Monorepo for an AI Incident Monitoring and Analysis system. Detects, evaluates, fingerprints, and forms engineering-ready incidents from AI agent execution telemetry, then generates GitHub Issues with root cause analysis. | ||
|
|
||
| **Runtime:** TypeScript (Node.js, Express, BullMQ) + Python (FastAPI, Gemini, PydanticAI) | ||
| **Infrastructure:** PostgreSQL 16, Redis 7, Docker Compose | ||
|
|
||
| --- | ||
|
|
||
| ## 📁 Monorepo Structure | ||
| ## Monorepo Structure | ||
|
|
||
| ``` | ||
| ```text | ||
| void-server/ | ||
| ├── docker-compose.yml # PostgreSQL 16, Redis 7, Adminer | ||
| ├── turbo.json # Turborepo task runner configuration | ||
| ├── package.json # npm workspaces root configuration | ||
| ├── .env.example # Environment variables template | ||
| ├── docker-compose.yml # PostgreSQL 16, Redis 7, Adminer | ||
| ├── turbo.json # Turborepo task runner | ||
| ├── package.json # npm workspaces root | ||
| │ | ||
| ├── apps/ | ||
| │ ├── node-api/ # Node.js TypeScript + Express API (Port 3001) | ||
| │ │ └── src/index.ts # Express server & Prisma database queries | ||
| │ ├── node-api/ # Express API (port 3001) + BullMQ worker | ||
| │ │ └── src/ | ||
| │ │ ├── index.ts # Express server: /health, /api/incidents | ||
| │ │ └── worker.ts # BullMQ worker: spawns Python evaluator subprocess | ||
| │ │ | ||
| │ └── fastapi-api/ # Python FastAPI Service (Port 8000) | ||
| │ ├── main.py # FastAPI app & endpoints | ||
| │ ├── database.py # SQLAlchemy session setup | ||
| │ ├── models.py # Incidents & Reports ORM models | ||
| │ └── requirements.txt | ||
| │ └── fastapi-api/ # FastAPI service (port 8000), SQLAlchemy | ||
| │ ├── main.py # /health, /api/incidents, /api/reports | ||
| │ ├── database.py # SQLAlchemy session | ||
| │ └── models.py # Incident & Report ORM models | ||
| │ | ||
| └── packages/ | ||
| ├── db/ # Shared Database Package (Prisma ORM) | ||
| │ └── prisma/schema.prisma # Incidents 1:N Reports schema definition | ||
| │ | ||
| ├── risk-engine/ # Deterministic risk evaluation engine | ||
| │ └── src/ | ||
| │ ├── evaluator.ts # Orchestrator: 8 policy checks → risk labels | ||
| │ ├── types.ts # RiskLabel enum (10 values), Execution, config types | ||
| │ └── policies/ # 8 pure policy functions (latency, tokens, crashes...) | ||
| │ | ||
| ├── incident-fingerprint/ # SHA-256 incident fingerprinting | ||
| │ └── src/ | ||
| │ ├── risk-labels.ts # normalizeRiskLabels() — validate, dedup, sort | ||
| │ ├── incident-fingerprint.ts # generateFingerprint() — SHA-256 hex hash | ||
| │ └── types.ts # RiskLabel enum (10 values), Severity type | ||
| │ | ||
| └── incident-formation/ # Incident persistence + BullMQ queue | ||
| └── src/ | ||
| ├── service.ts # IncidentFormationService — severity-routed orchestration | ||
| ├── repository.ts # PrismaIncidentRepository — CRUD via fingerprint | ||
| ├── queue.ts # BullMqIncidentQueue — Redis-backed, stable jobId | ||
| └── types.ts # IncidentInput, ProcessResult, interfaces | ||
| ├── packages/ | ||
| │ ├── db/ # Shared Prisma client + schema | ||
| │ │ └── prisma/schema.prisma # Incident 1:N Report, JSONB fields | ||
| │ │ | ||
| │ ├── risk-engine/ # Deterministic risk evaluation (TS) | ||
| │ │ └── src/ | ||
| │ │ ├── evaluator.ts # Orchestrator: 8 policy checks → labels | ||
| │ │ ├── types.ts # RiskLabel enum (10 values) | ||
| │ │ └── policies/ # 8 policies (latency, tokens, crashes...) | ||
| │ │ | ||
| │ ├── incident-fingerprint/ # SHA-256 dedup fingerprinting (TS) | ||
| │ │ └── src/ | ||
| │ │ ├── risk-labels.ts # normalizeRiskLabels() — dedup, sort, validate | ||
| │ │ └── incident-fingerprint.ts # generateFingerprint() | ||
| │ │ | ||
| │ ├── incident-formation/ # Persistence + BullMQ queue (TS) | ||
| │ │ └── src/ | ||
| │ │ ├── service.ts # Severity-routed orchestration | ||
| │ │ ├── repository.ts # PrismaIncidentRepository | ||
| │ │ └── queue.ts # BullMqIncidentQueue | ||
| │ │ | ||
| │ ├── adaptive-sampling/ # 1-in-N sampling of healthy executions (TS) | ||
| │ │ └── src/ | ||
| │ │ ├── service.ts # Window-based random sampling | ||
| │ │ └── queue.ts # BullMqSamplingQueue | ||
| │ │ | ||
| │ ├── evaluator/ # AI incident evaluator (Python/Gemini) | ||
| │ │ └── src/evaluator/ | ||
| │ │ ├── __main__.py # CLI: stdin → JSON | ||
| │ │ ├── agent.py # Orchestrator: context → prompt → Gemini → validate → score | ||
| │ │ ├── context_builder.py # Parse incident → EvaluationContext | ||
| │ │ ├── prompt_builder.py # Build Gemini prompt (v3) | ||
| │ │ ├── gemini_client.py # Gemini API call (gemini-3.1-flash-lite) | ||
| │ │ ├── validator.py # JSON parse + Pydantic validation | ||
| │ │ └── scorer.py # Deterministic confidence adjustments | ||
| │ │ | ||
| │ └── issue-agent/ # AI GitHub issue generator (Python/PydanticAI) | ||
| │ └── src/issue_agent/ | ||
| │ ├── __main__.py # CLI: stdin → EngineeringReport / GitHub Issue | ||
| │ ├── agent.py # PydanticAI Agent with 4 tools | ||
| │ ├── evidence.py # Extract failed tool calls from trace | ||
| │ ├── repository.py # GitHubRepo (prod) / LocalRepo (dev) | ||
| │ └── schemas.py # IncidentSnapshot, EngineeringReport (18 fields) | ||
| │ | ||
| └── evaluation_dataset/ # 16 incident scenarios for testing | ||
| └── incidents/ | ||
| ├── example/ | ||
| ├── context-overflow/ | ||
| ├── tool-anomaly/ | ||
| ├── looping/ | ||
| └── ... (16 total) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Pipeline | ||
|
|
||
| ```text | ||
| Agent Execution Telemetry | ||
| │ | ||
| ▼ | ||
| POST /api/traces | ||
| [apps/node-api] | ||
| │ | ||
| ▼ | ||
| Risk Engine (8 policies) | ||
| [packages/risk-engine] | ||
| │ | ||
| ▼ | ||
| normalizeRiskLabels() | ||
| [packages/incident-fingerprint] | ||
| │ | ||
| ▼ | ||
| IncidentFormationService | ||
| [packages/incident-formation] | ||
| │ | ||
| ┌────┴─────────┐ | ||
| │ │ | ||
| HEALTHY SUSPICIOUS | ||
| │ / CRITICAL | ||
| │ │ | ||
| ▼ ▼ | ||
| Adaptive persist | ||
| Sampling + queue | ||
| (1-in-N) [incident-analysis] | ||
| │ │ | ||
| ▼ ▼ | ||
| sampling- worker.ts | ||
| consumer.ts evaluator | ||
| (evaluator) → promotion gate | ||
| │ → issue agent | ||
| │ │ | ||
| ▼ ▼ | ||
| promotion? engineering report | ||
| │ + GitHub issue | ||
| ┌──┴──┐ | ||
| no yes | ||
| │ │ | ||
| skip IncidentFormationService | ||
| (SUSPICIOUS) | ||
| → incident-analysis queue | ||
| → worker.ts (same as above) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## 🗄️ Database Schema Design (`Incidents` 1 ➔ N `Reports`) | ||
| ## What We've Implemented | ||
|
|
||
| - **`incidents`**: `id`, `fingerprint` (unique), `trace_id`, `execution_id`, `title`, `severity`, `status`, `confidence`, `first_scene`, `last_scene`, `latest_report_id`, `occurrence`, `last_seen`, `analysis_status`, `latest_labels` (JSONB), `created_at`, `updated_at`. | ||
| | Layer | Package | Status | Details | | ||
| |---|---|---|---| | ||
| | **Risk Detection** | `risk-engine` | ✅ Complete | 8 deterministic policies, severity assignment, 90+ tests | | ||
| | **Fingerprinting** | `incident-fingerprint` | ✅ Complete | SHA-256 dedup, label normalization | | ||
| | **Incident Formation** | `incident-formation` | ✅ Complete | Severity routing, persistence, BullMQ queue, idempotency | | ||
| | **Adaptive Sampling** | `adaptive-sampling` | ✅ Complete | 1-in-N window sampling for healthy executions | | ||
| | **Database** | `@void-server/db` | ✅ Complete | Prisma schema, migrations, Incident 1:N Report | | ||
| | **Node API** | `node-api` | ✅ Complete | Express server, CRUD endpoints, BullMQ worker → Python subprocess | | ||
| | **FastAPI Service** | `fastapi-api` | ✅ Complete | Mirror API for Python-side access | | ||
| | **AI Evaluator** | `evaluator` (Python) | ✅ Complete | Gemini-powered failure mode detection, confidence scoring, 16-schema evaluation | | ||
| | **Issue Agent** | `issue-agent` (Python) | ✅ Complete | Repository investigation, code graph, timeline reconstruction, GitHub issue generation | | ||
| | **Test Scenarios** | `evaluation_dataset` | ✅ Complete | 16 incident scenarios covering all failure modes | | ||
| | **Infrastructure** | Docker Compose | ✅ Complete | PostgreSQL 16, Redis 7, Adminer, service Dockerfiles | | ||
| | **CI/CD** | turbo.json | ✅ Complete | Build pipeline, workspace orchestration | | ||
|
|
||
| --- | ||
|
|
||
| ## Project Stats | ||
|
|
||
| - **Languages:** TypeScript (~5,000 LOC) + Python (~3,500 LOC) | ||
| - **Packages:** 9 total (5 TS + 2 Python packages + 2 apps) | ||
| - **Tests:** ~180 unit/integration tests across all packages | ||
| - **Incident Evaluation Dataset:** 16 scenarios (example, context-overflow, handoff-failure, looping, tool-anomaly, crash-loop, critical-escalation, false-positive, insufficient-evidence, mixed-labels, rate-limit, recurring, transient-error, silent-hallucination, near-perfect, ambiguous-edge-case) | ||
| - **AI Models:** Google Gemini 3.1 Flash Lite (evaluator + issue agent) | ||
| - **Infrastructure:** PostgreSQL 16 (port 5435), Redis 7 (port 6379) | ||
|
|
||
| --- | ||
|
|
||
| ## Database Schema (`Incidents` 1:N `Reports`) | ||
|
|
||
| - **`incidents`**: `id`, `fingerprint` (unique), `trace_id`, `execution_id`, `title`, `severity`, `status`, `confidence`, `first_scene`, `last_scene`, `latest_report_id`, `occurrence`, `last_seen`, `analysis_status` (PENDING/PROCESSING/COMPLETED/FAILED), `latest_labels` (JSONB), `agent_steps` (JSON), `telemetry` (JSON), `engineering_report` (JSONB), `issue_url`, `created_at`, `updated_at`. | ||
| - **`reports`**: `id`, `incident_id` (FK), `model`, `report` (JSONB), `generated_at`. | ||
|
|
||
| --- | ||
|
|
||
| ## 🔄 Pipeline | ||
| ## Incident Formation Rules | ||
|
|
||
| ``` | ||
| Risk Evaluation Result | ||
| ↓ | ||
| normalizeRiskLabels() [packages/incident-fingerprint] | ||
| ↓ | ||
| generateFingerprint() [packages/incident-fingerprint] | ||
| ↓ | ||
| IncidentFormationService [packages/incident-formation] | ||
| ↓ | ||
| ┌────┼────┐ | ||
| │ │ │ | ||
| HEALTHY SUSPICIOUS CRITICAL | ||
| │ │ │ | ||
| skip persist persist | ||
| queue queue | ||
| "evaluate-incident" "critical-incident" | ||
| ``` | ||
| | Severity | Persisted? | Queued? | Job Name | | ||
| |---|---|---|---| | ||
| | HEALTHY | No (sampled 1-in-N) | Sampled to queue | `adaptive-sampling` | | ||
| | SUSPICIOUS | Yes | Yes (on creation) | `evaluate-incident` | | ||
| | CRITICAL | Yes | Yes (on creation / escalation) | `critical-incident` | | ||
|
|
||
| --- | ||
|
|
||
| ## ⚡ Quickstart | ||
| ## Quickstart | ||
|
|
||
| ### 1. Configure Environment & Start Infrastructure | ||
|
|
||
| ```bash | ||
| # Create local environment config | ||
| cp .env.example .env | ||
|
|
||
| # Start database & Redis infrastructure for local dev | ||
| npm run db:up | ||
| ``` | ||
| - **PostgreSQL**: `localhost:5435` (User: `void`, Pass: `voidpass`, DB: `void_db`) | ||
| - **Redis**: `localhost:6379` | ||
| - **Adminer**: [http://localhost:8088](http://localhost:8088) | ||
|
|
||
| ### 2. Run Database Migrations / Push Schema | ||
| - PostgreSQL: `localhost:5435` (User: `void`, Pass: `voidpass`, DB: `void_db`) | ||
| - Redis: `localhost:6379` | ||
| - Adminer: http://localhost:8088 | ||
|
|
||
| ### 2. Database Migrations | ||
|
|
||
| ```bash | ||
| npm run db:push | ||
| ``` | ||
|
|
||
| ### 3. Run Development Servers | ||
| ```bash | ||
| # Run all workspaces via Turborepo | ||
| npm run dev | ||
|
|
||
| # Or run specific workspace | ||
| npm run dev --workspace=@void-server/node-api | ||
|
|
||
| # Or run FastAPI Service (Port 8000) | ||
| cd apps/fastapi-api && pip install -r requirements.txt && python main.py | ||
| ```bash | ||
| npm run dev # All workspaces via Turborepo | ||
| npm run dev --workspace=@void-server/node-api # Node API only (port 3001) | ||
| cd apps/fastapi-api && pip install -r requirements.txt && python main.py # FastAPI (port 8000) | ||
| ``` | ||
|
|
||
| ### 4. Run Tests | ||
| ### 4. Run All Tests | ||
|
|
||
| ```bash | ||
| # All packages | ||
| # TypeScript packages (vitest) | ||
| npm test --workspace=@void-server/risk-engine | ||
| npm test --workspace=@void-server/incident-fingerprint | ||
| npm test --workspace=@void-server/incident-formation | ||
| npm test --workspace=@void-server/adaptive-sampling | ||
|
|
||
| # Python packages (unittest) — issue agent | ||
| PYTHONPATH=packages/issue-agent/src:packages/evaluator/src:packages/issue-agent/tests \ | ||
| python3 -m unittest discover -s packages/issue-agent/tests -p "test_*.py" -v | ||
| ``` | ||
|
|
||
| ### 5. Issue Agent E2E Monitoring | ||
|
|
||
| ```bash | ||
| # Requires GOOGLE_API_KEY in .env | ||
|
|
||
| # Install dependencies first | ||
| pip install -e packages/evaluator -e packages/issue-agent | ||
| python3 packages/issue-agent/tests/e2e_monitor.py | ||
|
|
||
| # Run all 14 scenarios (produces full report per scenario) | ||
| packages/evaluator/.venv/bin/python3 packages/issue-agent/tests/e2e_monitor.py | ||
|
|
||
| # Run specific scenario(s) | ||
| packages/evaluator/.venv/bin/python3 packages/issue-agent/tests/e2e_monitor.py example tool-anomaly | ||
|
|
||
| # Run issue agent unit tests | ||
| PYTHONPATH=packages/issue-agent/src:packages/evaluator/src:packages/issue-agent/tests python3 -m unittest packages/issue-agent/tests/test_schemas.py packages/issue-agent/tests/test_agent.py packages/issue-agent/tests/test_repository.py packages/issue-agent/tests/test_mapper.py -v | ||
| # Run specific scenarios | ||
| python3 packages/issue-agent/tests/e2e_monitor.py example tool-anomaly | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## 🧩 Packages | ||
| ### 6. Run Evaluator CLI | ||
|
|
||
| | Package | Responsibility | | ||
| |---|---| | ||
| | `@void-server/risk-engine` | Evaluates executions against 8 deterministic policies → risk labels | | ||
| | `@void-server/incident-fingerprint` | Normalizes labels, generates SHA-256 fingerprint | | ||
| | `@void-server/incident-formation` | Persists incidents, queues analysis via BullMQ | | ||
| | `@void-server/db` | Prisma client, shared database types | | ||
| ```bash | ||
| echo '{"id":"test","execution_id":"test","trace_id":"test","severity":"SUSPICIOUS","status":"OPEN","confidence":0,"occurrence":1,"analysis_status":"PENDING","agent_steps":[],"telemetry":null}' | \ | ||
| PYTHONPATH=packages/evaluator/src python3 -m evaluator | ||
| ``` | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| ### Incident Formation Rules | ||
| ### 7. Run Issue Agent CLI (Dev Mode) | ||
|
|
||
| | Severity | Persisted? | Queued? | Job Name | | ||
| |---|---|---|---| | ||
| | HEALTHY | No | No | — | | ||
| | SUSPICIOUS | Yes | Yes (on creation) | `evaluate-incident` | | ||
| | CRITICAL | Yes | Yes (on creation / escalation) | `critical-incident` | | ||
| ```bash | ||
| cat evaluation_dataset/incidents/example/incident.json | \ | ||
| PYTHONPATH=packages/issue-agent/src VOID_DEV_MODE=1 python3 -m issue_agent | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.