feat: Wire @l9/llm-router into Website Factory Bot - #1
Conversation
- Add packages/llm-router as workspace dependency (multi-provider model matrix) - Create src/services/llm.ts wrapper with convenience methods for all generation tasks - Add scripts/verify-visual-qa.mjs for LLM vision-based layout validation - Add contracts/llm_router_integration.yaml defining the integration interface - Update Makefile with 5 new targets (build-router, verify-launch-env, verify-visual-qa, generate-domain-spec, generate-content) - Update config/launch-env.required.yaml with llm_intelligence group - Rewrite DEPLOYMENT.md to match current env surface - Update VALIDATION.md with Launch Env and Visual QA classes - Add L9_META headers to new files - Update README, AGENTS.md, ARCHITECTURE.md for router integration - Run l9-recursive-optimization (score: 72 → 94, 0 violations remaining) Providers: OpenRouter (GPT-4o, Claude, Gemini) + Perplexity (Sonar, Deep Research) Budget: $200/mo per client, surge-aware, trajectory-based throttling Vision QA: Screenshot capture at 5 viewports → LLM layout validation
📝 WalkthroughWalkthroughIntroduces the ChangesLLM Router Package and Website Factory Integration
Sequence Diagram(s)sequenceDiagram
participant WebsiteFactoryLLM
participant L9LLMRouter
participant BudgetTracker
participant PerplexityClient
participant OpenRouterClient
WebsiteFactoryLLM->>L9LLMRouter: execute(task, systemPrompt, userPrompt)
L9LLMRouter->>L9LLMRouter: route(task) — select Perplexity/Vision/General
L9LLMRouter->>BudgetTracker: evaluateTask(clientId, task, estimatedCost)
BudgetTracker-->>L9LLMRouter: ThrottleDecision (allow/defer/downgrade)
alt Budget exhausted
L9LLMRouter-->>WebsiteFactoryLLM: throw BudgetExhaustedError
end
alt Search task (Perplexity)
L9LLMRouter->>PerplexityClient: complete / completeWithConsensus
PerplexityClient-->>L9LLMRouter: LLMResponse + citations
else Vision task (OpenRouter)
L9LLMRouter->>OpenRouterClient: completeWithVision(config, imageUrls)
OpenRouterClient-->>L9LLMRouter: LLMResponse
else General task (OpenRouter)
L9LLMRouter->>OpenRouterClient: complete / completeWithFallback
OpenRouterClient-->>L9LLMRouter: LLMResponse
end
L9LLMRouter->>BudgetTracker: recordSpend(clientId, actualCost)
L9LLMRouter-->>WebsiteFactoryLLM: LLMResponse.content
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.github.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates a new workspace package, @l9/llm-router, to provide centralized, budget-aware LLM routing and visual QA capabilities across the website generation pipeline. It updates documentation, environment configurations, and the build system to support this integration. The review feedback identifies several critical issues that must be addressed before merging: a bug in the router's execution method that bypasses model downgrades, TypeScript compilation errors due to type mismatches and incorrect property names in budget and visual QA configurations, and a runtime crash in the visual QA script caused by importing a TypeScript file directly into an ES module.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| async execute( | ||
| task: TaskDescriptor, | ||
| systemPrompt: string, | ||
| userPrompt: string, | ||
| options?: { | ||
| images?: string[]; | ||
| assistantContext?: string; | ||
| consensus?: boolean; | ||
| }, | ||
| ): Promise<LLMResponse> { | ||
| const decision = this.route(task); | ||
|
|
||
| // Check budget | ||
| const throttle = this.budget.evaluateTask( | ||
| task.clientId, | ||
| task, | ||
| decision.estimatedCost, | ||
| ); | ||
|
|
||
| if (!throttle.allowTask) { | ||
| throw new BudgetExhaustedError( | ||
| `Task deferred: ${throttle.reason}`, | ||
| task, | ||
| decision, | ||
| ); | ||
| } | ||
|
|
||
| // Apply model downgrade if throttled | ||
| if (throttle.forceDowngrade) { | ||
| decision.downgraded = true; | ||
| decision.downgradedFrom = decision.model; | ||
| decision.model = this.getDowngradedModel(decision.model, throttle.maxModelTier); | ||
| } | ||
|
|
||
| // Execute based on provider | ||
| let response: LLMResponse; | ||
|
|
||
| if (decision.provider === Provider.PERPLEXITY) { | ||
| const config = resolvePerplexityConfig(task); | ||
|
|
||
| if (options?.consensus && config.variations > 1) { | ||
| const result = await this.perplexity.completeWithConsensus( | ||
| config, | ||
| systemPrompt, | ||
| userPrompt, | ||
| options.assistantContext, | ||
| ); | ||
| response = result.best; | ||
| } else { | ||
| response = await this.perplexity.complete( | ||
| config, | ||
| systemPrompt, | ||
| userPrompt, | ||
| options?.assistantContext, | ||
| ); | ||
| } | ||
| } else if (VISION_TASK_TYPES.has(task.type) && options?.images?.length) { | ||
| const visionConfig = resolveVisionConfig( | ||
| task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION, | ||
| task.complexity, | ||
| options.images.length, | ||
| ); | ||
| response = await this.openrouter.completeWithVision( | ||
| visionConfig, | ||
| systemPrompt, | ||
| userPrompt, | ||
| options.images, | ||
| ); | ||
| } else { | ||
| const config = resolveGeneralConfig(task); | ||
| const fallbacks = getFallbackChain(config.model); | ||
| response = await this.openrouter.completeWithFallback( | ||
| config, | ||
| fallbacks, | ||
| systemPrompt, | ||
| userPrompt, | ||
| ); | ||
| } | ||
|
|
||
| // Record spend | ||
| this.budget.recordSpend(task.clientId, response.cost); | ||
|
|
||
| // Log the routing decision | ||
| decision.actualCost = response.cost; | ||
| decision.latencyMs = response.latencyMs; | ||
| this.callLog.push(decision); | ||
|
|
||
| return response; | ||
| } |
There was a problem hiding this comment.
There are two critical issues in the execute method:
- Downgrade Throttling Bypass: The downgrade throttling logic updates
decision.modelbut then resolves the config usingresolveGeneralConfig(task)orresolvePerplexityConfig(task)which only takestaskand does not know about the downgraded model. This means the actual API call is still made with the original expensive model, completely bypassing the downgrade mechanism. - Client ID Type Mismatch:
task.clientIdis optional (string | undefined) inTaskDescriptor, butRoutingDecision.clientIdis required (string). Assigningtask.clientIddirectly toclientIdinRoutingDecisionand passing it toevaluateTaskwithout a fallback will cause a TypeScript compilation error understrict: trueand potential runtime crashes.
We can resolve both issues by defaulting clientId to 'default' and overriding the resolved config's model with decision.model before executing the API call.
async execute(
task: TaskDescriptor,
systemPrompt: string,
userPrompt: string,
options?: {
images?: string[];
assistantContext?: string;
consensus?: boolean;
},
): Promise<LLMResponse> {
const clientId = task.clientId ?? 'default';
const decision = this.route({ ...task, clientId });
// Check budget
const throttle = this.budget.evaluateTask(
clientId,
task,
decision.estimatedCost,
);
if (!throttle.allowTask) {
throw new BudgetExhaustedError(
`Task deferred: ${throttle.reason}`,
task,
decision,
);
}
// Apply model downgrade if throttled
if (throttle.forceDowngrade) {
decision.downgraded = true;
decision.downgradedFrom = decision.model;
decision.model = this.getDowngradedModel(decision.model, throttle.maxModelTier);
}
// Execute based on provider
let response: LLMResponse;
if (decision.provider === Provider.PERPLEXITY) {
const config = resolvePerplexityConfig(task);
config.model = decision.model as SonarModel;
if (options?.consensus && config.variations > 1) {
const result = await this.perplexity.completeWithConsensus(
config,
systemPrompt,
userPrompt,
options.assistantContext,
);
response = result.best;
} else {
response = await this.perplexity.complete(
config,
systemPrompt,
userPrompt,
options?.assistantContext,
);
}
} else if (VISION_TASK_TYPES.has(task.type) && options?.images?.length) {
const visionConfig = resolveVisionConfig(
task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION,
task.complexity,
options.images.length,
);
visionConfig.model = decision.model as GeneralModel;
response = await this.openrouter.completeWithVision(
visionConfig,
systemPrompt,
userPrompt,
options.images,
);
} else {
const config = resolveGeneralConfig(task);
config.model = decision.model as GeneralModel;
const fallbacks = getFallbackChain(config.model);
response = await this.openrouter.completeWithFallback(
config,
fallbacks,
systemPrompt,
userPrompt,
);
}
// Record spend
this.budget.recordSpend(clientId, response.cost);
// Log the routing decision
decision.actualCost = response.cost;
decision.latencyMs = response.latencyMs;
this.callLog.push(decision);
return response;
}| route(task: TaskDescriptor): RoutingDecision { | ||
| // Search tasks → Perplexity | ||
| if (SEARCH_TASK_TYPES.has(task.type)) { | ||
| const config = resolvePerplexityConfig(task); | ||
| return { | ||
| taskId: task.id ?? crypto.randomUUID(), | ||
| clientId: task.clientId, | ||
| taskType: task.type, | ||
| complexity: task.complexity, | ||
| provider: Provider.PERPLEXITY, | ||
| model: config.model, | ||
| estimatedCost: config.estimatedCostPerCall, | ||
| reason: config.resolutionReason, | ||
| timestamp: new Date().toISOString(), | ||
| }; | ||
| } | ||
|
|
||
| // Vision tasks → OpenRouter with vision model | ||
| if (VISION_TASK_TYPES.has(task.type)) { | ||
| const config = resolveVisionConfig( | ||
| task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION, | ||
| task.complexity, | ||
| ); | ||
| return { | ||
| taskId: task.id ?? crypto.randomUUID(), | ||
| clientId: task.clientId, | ||
| taskType: task.type, | ||
| complexity: task.complexity, | ||
| provider: Provider.OPENROUTER, | ||
| model: config.model, | ||
| estimatedCost: config.estimatedCostPerCall, | ||
| reason: config.resolutionReason, | ||
| timestamp: new Date().toISOString(), | ||
| }; | ||
| } | ||
|
|
||
| // Everything else → OpenRouter general matrix | ||
| const config = resolveGeneralConfig(task); | ||
| return { | ||
| taskId: task.id ?? crypto.randomUUID(), | ||
| clientId: task.clientId, | ||
| taskType: task.type, | ||
| complexity: task.complexity, | ||
| provider: Provider.OPENROUTER, | ||
| model: config.model, | ||
| estimatedCost: config.estimatedCostPerCall, | ||
| reason: config.resolutionReason, | ||
| timestamp: new Date().toISOString(), | ||
| }; | ||
| } |
There was a problem hiding this comment.
Since task.clientId is optional (string | undefined) in TaskDescriptor, but RoutingDecision.clientId is required (string), assigning task.clientId directly to clientId in RoutingDecision will cause a TypeScript compilation error under strict: true.
We should fallback to 'default' if task.clientId is undefined.
route(task: TaskDescriptor): RoutingDecision {
const clientId = task.clientId ?? 'default';
// Search tasks → Perplexity
if (SEARCH_TASK_TYPES.has(task.type)) {
const config = resolvePerplexityConfig(task);
return {
taskId: task.id ?? crypto.randomUUID(),
clientId,
taskType: task.type,
complexity: task.complexity,
provider: Provider.PERPLEXITY,
model: config.model,
estimatedCost: config.estimatedCostPerCall,
reason: config.resolutionReason,
timestamp: new Date().toISOString(),
};
}
// Vision tasks → OpenRouter with vision model
if (VISION_TASK_TYPES.has(task.type)) {
const config = resolveVisionConfig(
task.type as TaskType.VISUAL_QA | TaskType.SCREENSHOT_ANALYSIS | TaskType.LAYOUT_VALIDATION,
task.complexity,
);
return {
taskId: task.id ?? crypto.randomUUID(),
clientId,
taskType: task.type,
complexity: task.complexity,
provider: Provider.OPENROUTER,
model: config.model,
estimatedCost: config.estimatedCostPerCall,
reason: config.resolutionReason,
timestamp: new Date().toISOString(),
};
}
// Everything else → OpenRouter general matrix
const config = resolveGeneralConfig(task);
return {
taskId: task.id ?? crypto.randomUUID(),
clientId,
taskType: task.type,
complexity: task.complexity,
provider: Provider.OPENROUTER,
model: config.model,
estimatedCost: config.estimatedCostPerCall,
reason: config.resolutionReason,
timestamp: new Date().toISOString(),
};
}| budget: { | ||
| monthlyBudgetPerClient: config.monthlyBudget ?? 200, | ||
| weeklyTarget: config.weeklyBudget ?? 50, | ||
| weeklyCeiling: (config.weeklyBudget ?? 50) * 2, | ||
| globalHardCeiling: 500, | ||
| surgeAllowance: true, | ||
| }, |
There was a problem hiding this comment.
The budget object literal passed to RouterConfig has incorrect property names (weeklyCeiling instead of weeklyHardCeiling, globalHardCeiling instead of globalMonthlyHardCeiling, and surgeAllowance instead of surgeThreshold). This will fail TypeScript compilation under strict: true.
We should use the correct property names from BudgetConfig.
| budget: { | |
| monthlyBudgetPerClient: config.monthlyBudget ?? 200, | |
| weeklyTarget: config.weeklyBudget ?? 50, | |
| weeklyCeiling: (config.weeklyBudget ?? 50) * 2, | |
| globalHardCeiling: 500, | |
| surgeAllowance: true, | |
| }, | |
| budget: { | |
| monthlyBudgetPerClient: config.monthlyBudget ?? 200, | |
| weeklyTarget: config.weeklyBudget ?? 50, | |
| weeklyHardCeiling: (config.weeklyBudget ?? 50) * 2, | |
| globalMonthlyHardCeiling: 500, | |
| surgeThreshold: 0.6, | |
| }, |
| planFullSiteQA(siteUrl: string, pages: string[]) { | ||
| const config: FullSiteQAConfig = { | ||
| siteUrl, | ||
| pages, | ||
| includeCompetitorComparison: true, | ||
| includeConversionAudit: true, | ||
| }; | ||
| return this.router.planVisualQA(config); | ||
| } |
There was a problem hiding this comment.
In planFullSiteQA, the config object passed to this.router.planVisualQA has incorrect property names (siteUrl, includeCompetitorComparison, includeConversionAudit) and misses required properties (viewports, conversionAudit). This will fail TypeScript compilation under strict: true.
We should use the correct property names from FullSiteQAConfig.
| planFullSiteQA(siteUrl: string, pages: string[]) { | |
| const config: FullSiteQAConfig = { | |
| siteUrl, | |
| pages, | |
| includeCompetitorComparison: true, | |
| includeConversionAudit: true, | |
| }; | |
| return this.router.planVisualQA(config); | |
| } | |
| planFullSiteQA(pages: string[], competitorUrl?: string) { | |
| const viewports = [ | |
| this.router.getViewports().desktop_1440, | |
| this.router.getViewports().mobile_iphone, | |
| ]; | |
| const config: FullSiteQAConfig = { | |
| pages, | |
| viewports, | |
| competitorUrl, | |
| conversionAudit: true, | |
| }; | |
| return this.router.planVisualQA(config); | |
| } |
| if (process.env.OPENROUTER_API_KEY) { | ||
| console.log('\n🤖 Running vision analysis...\n'); | ||
| // Dynamic import of the LLM service | ||
| const { createWebsiteFactoryLLM } = await import('../src/services/llm.ts'); |
There was a problem hiding this comment.
The script imports ../src/services/llm.ts directly. Since Node.js runs this .mjs file directly, it cannot resolve or transpile .ts files at runtime, leading to a crash. It should import from the compiled output ../dist/src/services/llm.js.
| const { createWebsiteFactoryLLM } = await import('../src/services/llm.ts'); | |
| const { createWebsiteFactoryLLM } = await import('../dist/src/services/llm.js'); |
![chatgpt-codex-connector[bot]](https://avatars.github.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34691705bf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| GPT4O_VISION = 'openai/gpt-4o', | ||
| CLAUDE_SONNET_VISION = 'anthropic/claude-sonnet-4', | ||
| GEMINI_FLASH_VISION = 'google/gemini-2.5-flash', |
There was a problem hiding this comment.
These vision enum members reuse the same string values as GPT4O, CLAUDE_SONNET, and GEMINI_FLASH above. When those enum members are used as computed keys in the router maps, TypeScript sees duplicate object-literal properties and npm run build:router fails with TS1117 before emitting dist, so consumers of the new workspace package cannot import it. Give the aliases distinct keys or remove the duplicate map entries.
Useful? React with 👍 / 👎.
| if (throttle.forceDowngrade) { | ||
| decision.downgraded = true; | ||
| decision.downgradedFrom = decision.model; | ||
| decision.model = this.getDowngradedModel(decision.model, throttle.maxModelTier); |
There was a problem hiding this comment.
When a client is in soft/hard throttle, this only changes the audit decision; immediately afterwards each branch rebuilds a fresh config with resolvePerplexityConfig, resolveVisionConfig, or resolveGeneralConfig, so the original expensive model is still sent to the provider while the log says it was downgraded. This defeats the budget enforcement exactly for over-budget clients; pass the downgraded model into the provider config before executing.
Useful? React with 👍 / 👎.
| weeklyCeiling: (config.weeklyBudget ?? 50) * 2, | ||
| globalHardCeiling: 500, | ||
| surgeAllowance: true, |
There was a problem hiding this comment.
These keys are not read by BudgetTracker, which only merges weeklyHardCeiling, globalMonthlyHardCeiling, and surgeThreshold. In a deployment that sets WEEKLY_BUDGET_TARGET, the weekly target changes but the hard ceiling stays at the default $100 and the global cap stays $2000, so operator budget overrides are silently not enforced.
Useful? React with 👍 / 👎.
| const issues = []; | ||
| for (const screenshot of screenshots) { | ||
| const result = await llm.validateLayout( | ||
| [screenshot.path], |
There was a problem hiding this comment.
captureScreenshots stores local filesystem paths, and this passes those paths straight to validateLayout; the provider client treats non-data: entries as remote image_url values, so OpenRouter cannot fetch /workspace/...png when visual QA runs with Playwright and OPENROUTER_API_KEY. Convert the files to data URLs or serve/upload them before invoking the vision model.
Useful? React with 👍 / 👎.
| const config: FullSiteQAConfig = { | ||
| siteUrl, | ||
| pages, | ||
| includeCompetitorComparison: true, | ||
| includeConversionAudit: true, |
There was a problem hiding this comment.
FullSiteQAConfig requires viewports and conversionAudit, but this wrapper passes siteUrl, includeCompetitorComparison, and includeConversionAudit instead. Calling planFullSiteQA() therefore reaches for (const viewport of config.viewports) with undefined and throws before any QA tasks are generated.
Useful? React with 👍 / 👎.
|
|
||
| // Determine variations (consensus mode from Enrichment Engine) | ||
| let variations = 1; | ||
| if (task.complexity >= TaskComplexity.HIGH) variations = 3; |
There was a problem hiding this comment.
TaskComplexity values are strings, so this comparison is lexicographic: low and medium compare greater than high, while critical does not. Low/medium Perplexity tasks therefore run three consensus variations and some critical/high routing checks are skipped, causing unexpected cost and model selection; use an explicit numeric rank for tier comparisons.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Note
Due to the large number of review comments, Critical severity comments were prioritized as inline comments.
🟠 Major comments (20)
Makefile-91-92 (1)
91-92:⚠️ Potential issue | 🟠 Major | ⚡ Quick winEnforce Visual QA and launch-env gates before production deploy.
Line 91-92 currently allows
make deploy-productionwithout the requiredverify:visual-qagate. This weakens the documented preview-first/evidence-backed release control path.Proposed fix
-deploy-production: +deploy-production: verify-launch-env verify-visual-qa npm run deploy:productionBased on learnings: "Visual QA verification (verify:visual-qa) is required before production deployment" and "Readiness claims must be evidence-backed only."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@Makefile` around lines 91 - 92, The deploy-production target in the Makefile currently executes directly without enforcing required quality gates. Add verify:visual-qa (and launch-env as mentioned in the requirement title) as prerequisites to the deploy-production target to enforce the preview-first/evidence-backed release control path. In Makefile syntax, list these prerequisites on the target declaration line before the colon, separated by spaces, so they must complete successfully before the npm run deploy:production command executes.Source: Learnings
.env.example-6-7 (1)
6-7:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUse empty values for operator-owned secrets in
.env.example.Line 6-7 pre-populates API keys with fake token-shaped values. This can accidentally satisfy non-empty validation and produce false “ready” status.
Proposed fix
-PERPLEXITY_API_KEY=pplx-xxxxxxxxxxxxxxxxxxxx -OPENROUTER_API_KEY=sk-or-xxxxxxxxxxxxxxxxxxxx +PERPLEXITY_API_KEY= +OPENROUTER_API_KEY=As per coding guidelines: "Convert missing operator-owned values into environment variables and enforce fail-closed validation using .env.example and config/launch-env.required.yaml." Based on learnings: "Record Unknowns rather than inventing values."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.env.example around lines 6 - 7, The PERPLEXITY_API_KEY and OPENROUTER_API_KEY environment variables in the .env.example file are populated with fake token-shaped placeholder values that can falsely satisfy non-empty validation checks, making the system appear ready when it lacks actual credentials. Replace these fake values with empty strings to ensure that validation properly fails when real operator-owned secrets are not configured, following the fail-closed validation principle and avoiding the practice of inventing placeholder values.Sources: Coding guidelines, Learnings
DEPLOYMENT.md-37-43 (1)
37-43:⚠️ Potential issue | 🟠 Major | ⚡ Quick winEnv variable naming and launch-requirement scope misalignment (root cause of V003 incomplete resolution).
The core issue:
DEPLOYMENT.mdlines 37–43 ("Site Runtime — required for launch") lists variables that either do not match names inconfig/launch-env.required.yamlor are not required for launch per the contract.Root cause: When violations V003 was marked FIXED (docs/recursive_optimization_report.md line 86), the stale variable names and scope were not fully corrected.
DEPLOYMENT.mdstill contains:
DEPLOYMENT.md#L37-L43:
PUBLIC_FORM_ENDPOINTshould beFORM_ENDPOINT_URL(matches form group in contract).PUBLIC_ANALYTICS_IDshould beANALYTICS_MEASUREMENT_ID(matches analytics group).ACCULYNX_API_KEYis listed as "required for launch," butconfig/launch-env.required.yamlmarks acculynx group asrequired_for_launch: falseandrequired_for_crm_launch_claim: true.Impact: Operator follows incorrect deployment instructions (wrong variable names, incorrect launch requirements) and may not set required variables or may set variables with mismatched names.
Corrective actions:
DEPLOYMENT.md#L37-L43: Either (a) relabel "Site Runtime" as "Site Runtime (required when features are claimed)" and list only truly required-for-launch variables, or (b) align all variable names and scope markers withconfig/launch-env.required.yaml.docs/recursive_optimization_report.md#L84-L88: Update the V003 FIXED statement to reflect the actual corrective action taken.As per coding guidelines, every readiness claim must map to file evidence or an explicit blocked-check record. The current state conflicts with the authoritative contract in
config/launch-env.required.yaml.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@DEPLOYMENT.md` around lines 37 - 43, The environment variable names and launch-requirement scope in DEPLOYMENT.md (lines 37–43) do not match the authoritative definitions in config/launch-env.required.yaml. Fix this in two places: (1) In DEPLOYMENT.md lines 37–43, correct the variable names: change PUBLIC_FORM_ENDPOINT to FORM_ENDPOINT_URL and PUBLIC_ANALYTICS_ID to ANALYTICS_MEASUREMENT_ID; clarify the section scope by either relabeling it to indicate conditional-feature requirements (not universal launch requirements) or remove ACCULYNX_API_KEY from the "required for launch" section since config/launch-env.required.yaml marks the acculynx group as required_for_launch: false. (2) In docs/recursive_optimization_report.md lines 84–88, update the V003 FIXED statement to reflect the actual corrective action taken (the name corrections and scope clarification in DEPLOYMENT.md).Source: Coding guidelines
contracts/llm_router_integration.yaml-57-57 (1)
57-57:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUse canonical provider tokens in the contract.
provider: openrouter (vision model)is not a stable provider identifier. Keepprovidercanonical (openrouter) and model/modality in a separate field so contract consumers can parse deterministically.Suggested normalization
- provider: openrouter (vision model) + provider: openrouter + modality: vision ... - provider: openrouter (vision model) + provider: openrouter + modality: visionAlso applies to: 66-66
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@contracts/llm_router_integration.yaml` at line 57, The provider field contains non-canonical provider identifiers that mix provider names with model/modality metadata. At line 57, change `provider: openrouter (vision model)` to use only the canonical provider name `openrouter` in the provider field, and add a separate field (such as `modality` or `variant`) to capture the vision model descriptor. Apply the same normalization at line 66 where a similar mixed provider identifier appears. This ensures contract consumers can parse the provider deterministically without string parsing.packages/llm-router/src/budget/index.ts-86-94 (1)
86-94:⚠️ Potential issue | 🟠 Major | ⚡ Quick winReject invalid spend amounts before mutating budget state.
recordSpendshould fail on negative or non-finite amounts; otherwise budget state can be reduced or corrupted, weakening throttle enforcement.Suggested guard
recordSpend(clientId: string, amount: number): void { + if (!Number.isFinite(amount) || amount < 0) { + throw new Error(`Invalid spend amount: ${amount}`); + } const state = this.getState(clientId);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/budget/index.ts` around lines 86 - 94, The recordSpend method does not validate the amount parameter before updating budget state, allowing negative or non-finite values to corrupt the budget tracking. Add a validation guard at the beginning of the recordSpend method that rejects negative amounts and non-finite values (NaN, Infinity) by throwing an error before any state mutations occur, ensuring only valid spend amounts are recorded.packages/llm-router/src/types.ts-16-17 (1)
16-17:⚠️ Potential issue | 🟠 Major | ⚡ Quick winRestrict
Providerto the supported runtime providers.Keeping
OPENAI_DIRECTandANTHROPIC_DIRECTin the public enum expands the contract beyond the locked integration and risks invalid routing states when only OpenRouter/Perplexity credentials are configured.Suggested contract-tightening diff
export enum Provider { OPENROUTER = 'openrouter', PERPLEXITY = 'perplexity', - OPENAI_DIRECT = 'openai_direct', - ANTHROPIC_DIRECT = 'anthropic_direct', }Based on learnings: "LLM Providers are locked to OpenRouter (general) and Perplexity (search-grounded); do not use other LLM providers."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/types.ts` around lines 16 - 17, Remove the OPENAI_DIRECT and ANTHROPIC_DIRECT enum values from the Provider enum in the types.ts file. These providers are not part of the locked integration (only OpenRouter and Perplexity are supported at runtime), and keeping them in the public contract exposes invalid routing states. Retain only the Provider enum values that correspond to the actually supported runtime providers.Source: Learnings
packages/llm-router/src/matrices/perplexity-matrix.ts-168-170 (1)
168-170:⚠️ Potential issue | 🟠 Major
TaskComplexityis a string enum whose lexical ordering contradicts its semantic ordering, breaking all relational comparisons. String values ('trivial','low','medium','high','critical') sort alphabetically ascritical < high < low < medium < trivial, but semantic complexity ranks astrivial < low < medium < high < critical. This causes deterministic misrouting: e.g.,complexity >= TaskComplexity.MEDIUMwill incorrectly evaluate totruewhen complexity is'trivial'.Replace all relational comparisons with an explicit numeric rank map (e.g.,
TRIVIAL=0, LOW=1, MEDIUM=2, HIGH=3, CRITICAL=4):
packages/llm-router/src/matrices/perplexity-matrix.ts#L168-L170: use rank comparison for context sizepackages/llm-router/src/matrices/perplexity-matrix.ts#L180: use rank comparisonpackages/llm-router/src/matrices/general-matrix.ts#L213-L219: use rank comparison for token allocationpackages/llm-router/src/matrices/perplexity-matrix.ts#L226, L249, L290: apply same fix to remaining comparisons🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/matrices/perplexity-matrix.ts` around lines 168 - 170, The TaskComplexity string enum has lexical ordering that contradicts semantic ordering (alphabetically: critical < high < low < medium < trivial, but semantically: trivial < low < medium < high < critical), causing all relational comparisons to produce incorrect results. Create an explicit numeric rank map for TaskComplexity (mapping TRIVIAL=0, LOW=1, MEDIUM=2, HIGH=3, CRITICAL=4), then use rank comparisons instead of direct enum comparisons. In packages/llm-router/src/matrices/perplexity-matrix.ts lines 168-170, replace the relational comparison with a rank-based comparison for the context size determination. In packages/llm-router/src/matrices/general-matrix.ts lines 213-219, replace the relational comparisons used for token allocation with rank-based comparisons. Additionally, apply the same rank-based comparison fix to the remaining TaskComplexity comparisons in packages/llm-router/src/matrices/perplexity-matrix.ts at lines 180, 226, 249, and 290.packages/llm-router/src/types.ts-50-53 (1)
50-53:⚠️ Potential issue | 🟠 MajorDuplicate
GeneralModelenum values cause object key collisions, overwriting definitions. Vision variants (_VISION) share identical runtime string values with base models, so keying objects by enum values causes the second definition to overwrite the first.Affected locations:
packages/llm-router/src/types.ts#L50-L53: Vision enum members duplicate base model valuespackages/llm-router/src/matrices/general-matrix.ts#L183-L199and otherRecord<GeneralModel, ...>definitions in this file: fallback chains, cost maps, and provider model IDs are corrupted by key collisionsSolution: Assign unique string values to vision variants (e.g.,
'openai/gpt-4o-vision'instead of'openai/gpt-4o'), or redesign the enum to carry modality separately and avoid using colliding values as object keys.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/types.ts` around lines 50 - 53, The vision model enum members (GPT4O_VISION, CLAUDE_SONNET_VISION, GEMINI_FLASH_VISION) in packages/llm-router/src/types.ts#L50-L53 are assigned identical string values to their base model counterparts, causing object key collisions when used as Record keys. Assign unique string values to each vision variant by appending a suffix (e.g., change 'openai/gpt-4o' to 'openai/gpt-4o-vision' for GPT4O_VISION, 'anthropic/claude-sonnet-4' to 'anthropic/claude-sonnet-4-vision' for CLAUDE_SONNET_VISION, and 'google/gemini-2.5-flash' to 'google/gemini-2.5-flash-vision' for GEMINI_FLASH_VISION). After fixing the enum values, verify that the Record<GeneralModel, ...> definitions in packages/llm-router/src/matrices/general-matrix.ts#L183-L199 (fallback chains, cost maps, and provider model IDs) work correctly with the now-unique values; no direct changes should be needed at this location once the enum is corrected.packages/llm-router/src/budget/index.ts-146-149 (1)
146-149:⚠️ Potential issue | 🟠 Major | ⚡ Quick winBudget gating ignores projected spend (
estimatedCost).
evaluateTaskreceivesestimatedCostbut computes throttle from current spend only, so tasks can be approved even when they will immediately push weekly/monthly spend past hard limits.Suggested fix (projected-state evaluation)
evaluateTask(clientId: string, task: TaskDescriptor, estimatedCost: number): ThrottleDecision { const state = this.getState(clientId); - const throttle = this.computeThrottleLevel(state); + const projectedState: BudgetState = { + ...state, + monthSpend: state.monthSpend + estimatedCost, + weekSpend: state.weekSpend + estimatedCost, + todaySpend: state.todaySpend + estimatedCost, + remainingMonthly: state.monthlyBudget - (state.monthSpend + estimatedCost), + remainingWeekly: state.weeklyHardCeiling - (state.weekSpend + estimatedCost), + }; + const throttle = this.computeThrottleLevel(projectedState);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/budget/index.ts` around lines 146 - 149, The evaluateTask method receives an estimatedCost parameter that is currently unused when computing throttle decisions. To fix this, modify the method to compute throttle based on projected spend rather than current spend only. After calling getState to retrieve the current state, create a projected state by adding the estimatedCost to the current spend amounts, then pass this projected state to computeThrottleLevel instead of passing the current state. This ensures that throttle decisions account for the immediate impact of the task being evaluated.scripts/verify-visual-qa.mjs-133-135 (1)
133-135:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAlways persist
visual_qa_report.json, even when AI analysis is skipped.Lines 133-135 only log a message and do not emit a report artifact. That breaks deterministic verification evidence for this gate.
Based on learnings, readiness/verification claims should be evidence-backed and visual QA verification is required before production decisions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verify-visual-qa.mjs` around lines 133 - 135, The else block at lines 133-135 only logs a message when AI analysis is skipped due to missing OPENROUTER_API_KEY, but does not persist the visual_qa_report.json file. This means no artifact is created to provide evidence of the verification. Modify the else branch to emit or write the visual_qa_report.json file (containing the screenshots data captured before this point) to disk, ensuring that verification evidence is always persisted regardless of whether AI analysis was performed or skipped.Source: Learnings
src/services/llm.ts-286-289 (1)
286-289:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFactory defaults are fail-open for tenant identity and budget config.
Line 286 falls back to a shared
'default'client ID, and Lines 287-288 silently coerce invalid numeric env values to defaults. This can collapse tenant budget/call-log isolation and hide launch-env misconfiguration.Based on learnings, unknown operator-owned values should be recorded and validated fail-closed, with launch checks enforced by
npm run verify:launch-env.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/services/llm.ts` around lines 286 - 289, The configuration fallback on lines 286-288 is fail-open: using a shared 'default' clientId and silently coercing invalid numeric environment values. Replace these lenient defaults with strict validation that throws an error if required values are missing or invalid. Remove the fallback to 'default' for clientId and instead require it to be explicitly provided; validate that MONTHLY_BUDGET_PER_CLIENT and WEEKLY_BUDGET_TARGET are valid numbers, throwing an error if they are missing or cannot be parsed. This validation should be enforced through the launch environment check process to catch misconfigurations during deployment rather than silently using defaults that could collapse tenant isolation.Source: Learnings
scripts/verify-visual-qa.mjs-45-48 (1)
45-48:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPreflight key check is inconsistent with the service factory contract.
Lines 45-48 only guard
OPENROUTER_API_KEY, but Lines 103-107 callcreateWebsiteFactoryLLM(), which throws unless both provider keys are set. This causes avoidable runtime failure during “vision analysis enabled” paths.Also applies to: 103-107
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verify-visual-qa.mjs` around lines 45 - 48, The preflight check at lines 45-48 only validates OPENROUTER_API_KEY, but the createWebsiteFactoryLLM() call at lines 103-107 requires both OPENROUTER_API_KEY and another provider key to be set. Update the guard condition at lines 45-48 to check for both keys that createWebsiteFactoryLLM() requires, so that missing dependencies are caught early with a clear message rather than causing a runtime failure when the factory is instantiated at lines 103-107. This ensures the preflight validation matches the actual contract of createWebsiteFactoryLLM().scripts/verify-visual-qa.mjs-180-180 (1)
180-180:⚠️ Potential issue | 🟠 Major | ⚡ Quick winTop-level error handling should fail the process.
Line 180 logs errors but leaves exit status successful, which can let verification failures pass CI gates.
Suggested fix
-main().catch(console.error); +main().catch((err) => { + console.error(err); + process.exitCode = 1; +});🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verify-visual-qa.mjs` at line 180, The catch handler for the main() promise at line 180 (main().catch(console.error)) only logs errors to console but does not exit the process with a failure status. Modify the error handler to call process.exit(1) after logging the error, ensuring that the process terminates with a non-zero exit code when main() rejects, which will properly signal verification failures to CI systems.scripts/verify-visual-qa.mjs-123-123 (1)
123-123:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFAIL/PASS gate is brittle due to case-sensitive free-text matching.
Line 123 checks only
includes('critical'); responses likeCriticalor structured severity fields can be missed, producing false PASS outcomes.Suggested fix
- status: issues.some(i => i.analysis.includes('critical')) ? 'FAIL' : 'PASS', + status: issues.some(i => /\bcritical\b/i.test(i.analysis)) ? 'FAIL' : 'PASS',🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verify-visual-qa.mjs` at line 123, The status check on line 123 uses case-sensitive string matching with includes('critical'), which fails to catch variations like 'Critical' or 'CRITICAL', leading to false PASS outcomes. Fix this by converting the analysis string to lowercase before checking if it includes the critical keyword, ensuring the FAIL/PASS gate properly catches all case variations of critical severity levels.scripts/verify-visual-qa.mjs-63-63 (1)
63-63:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUse argument-safe process execution and verify HTTP status explicitly.
Line 63 interpolates
SITE_URLinto a shell command and never checks whether the returned status code is successful. That both opens command-injection risk and treats HTTP 4xx/5xx as “reachable.”Suggested fix
-import { execSync } from 'node:child_process'; +import { execSync, execFileSync } from 'node:child_process'; @@ - try { - execSync(`curl -s -o /dev/null -w "%{http_code}" ${siteUrl}`, { encoding: 'utf-8' }); - } catch { + try { + const normalized = new URL(siteUrl).toString(); + const httpCode = execFileSync( + 'curl', + ['-s', '-o', '/dev/null', '-w', '%{http_code}', normalized], + { encoding: 'utf-8' }, + ).trim(); + if (!/^[23]\d\d$/.test(httpCode)) { + throw new Error(`HTTP ${httpCode}`); + } + } catch { console.error('❌ Site not reachable at', siteUrl);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verify-visual-qa.mjs` at line 63, The execSync call at line 63 interpolates siteUrl directly into a shell command string, creating a command-injection vulnerability, and does not verify that the HTTP status code indicates success. Replace the string interpolation with argument-safe process execution by passing the URL as a separate argument to curl, and explicitly check that the returned HTTP status code falls within the 2xx success range before proceeding. If the status code indicates failure (4xx, 5xx, etc.), throw an error to halt execution.packages/llm-router/src/index.ts-145-157 (1)
145-157:⚠️ Potential issue | 🟠 Major | ⚡ Quick winVision tasks should fail fast when images are missing.
At Line 145, if a vision task arrives without
options.images, execution falls through to the general text branch (Line 157). This silently routes a vision workload to non-vision generation instead of surfacing an input contract error.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/index.ts` around lines 145 - 157, Vision tasks are silently falling through to the text generation branch when images are not provided, instead of failing with a clear error. Add a validation check that detects when a task type is in VISION_TASK_TYPES but options.images is missing or empty, and throw an error with a descriptive message about the missing images before the condition on Line 145. This ensures vision task inputs are validated against their contract requirements and fail fast rather than producing incorrect results.packages/llm-router/src/providers/perplexity.ts-65-68 (1)
65-68:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
disableSearchis a no-op and search stays enabled.Lines 65-68 check
config.disableSearchbut apply no behavior, whileweb_search_optionsis still always set at Line 60. This makes search-disable requests ineffective.Suggested fix
- const requestBody: Record<string, unknown> = { + const requestBody: Record<string, unknown> = { model: config.model, messages, temperature: config.temperature, max_tokens: config.maxTokens, - web_search_options: { - search_context_size: config.searchContextSize, - }, }; - // Add response format if JSON - if (!config.disableSearch) { - // Search is enabled by default for Perplexity - } + if (!config.disableSearch) { + requestBody.web_search_options = { + search_context_size: config.searchContextSize, + }; + }Also applies to: 60-63
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/providers/perplexity.ts` around lines 65 - 68, The disableSearch configuration option is ineffective because web_search_options is unconditionally set at lines 60-63 in packages/llm-router/src/providers/perplexity.ts while the check at lines 65-68 applies no behavior. To fix this, move the web_search_options assignment at lines 60-63 inside the conditional block so it only gets set when config.disableSearch is false (meaning search is enabled). This ensures that when disableSearch is true, web_search_options is not applied and search remains disabled as intended. Remove or restructure the empty conditional at lines 65-68 after moving the logic.packages/llm-router/src/vision/index.ts-316-336 (1)
316-336:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift
generateFullSiteQAPlanpasses page URLs where screenshot image URLs are expected.At Lines 318/326/335,
pageandcompetitorUrlare forwarded into task builders whose parameters are explicitlyscreenshotUrl(Lines 242, 258, 274). This creates tasks that treat webpage URLs as vision image inputs, breaking execution semantics.Also applies to: 242-243, 258-259, 274-275
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/vision/index.ts` around lines 316 - 336, The task builders buildLayoutValidationTask, buildCompetitorComparisonTask, and buildConversionAuditTask expect screenshot image URLs as their first parameter (defined as screenshotUrl at lines 242-243, 258-259, and 274-275 respectively), but in the generateFullSiteQAPlan function, webpage URLs from config.pages and config.competitorUrl are being passed directly to these builders at lines 318, 326, and 335. Convert the webpage URLs to screenshot image URLs before passing them to each task builder call. Ensure that wherever page URLs are forwarded to these three task builders, they are first transformed into the corresponding screenshot URLs that the builders expect.packages/llm-router/src/vision/index.ts-192-205 (1)
192-205:⚠️ Potential issue | 🟠 Major | ⚡ Quick winResolver order blocks competitor branch selection.
At Line 192,
complexity >= TaskComplexity.HIGHreturns early, so the multi-image branch at Line 204 is skipped.buildCompetitorComparisonTask(Line 266) therefore never gets the intended competitor-specific model path.Suggested fix
- // For detailed layout validation, use GPT-4o (best at structured visual analysis) - if (taskType === TaskType.LAYOUT_VALIDATION || complexity >= TaskComplexity.HIGH) { + // For competitor comparison (multiple images), use Claude (best at nuanced comparison) + if (imageCount > 1) { + return { + model: GeneralModel.CLAUDE_SONNET_VISION, + provider: Provider.OPENROUTER, + maxTokens: 2048, + detail: 'high', + estimatedCostPerCall: 0.03, + resolutionReason: 'Multi-image comparison — Claude best at nuanced visual reasoning', + }; + } + + // For detailed layout validation, use GPT-4o (best at structured visual analysis) + if (taskType === TaskType.LAYOUT_VALIDATION || complexity >= TaskComplexity.HIGH) { return { model: GeneralModel.GPT4O_VISION, provider: Provider.OPENROUTER, maxTokens: 2048, detail: 'high', estimatedCostPerCall: 0.02, resolutionReason: 'Detailed layout validation — GPT-4o best structured visual analysis', }; } - - // For competitor comparison (multiple images), use Claude (best at nuanced comparison) - if (imageCount > 1) { - return { - model: GeneralModel.CLAUDE_SONNET_VISION, - provider: Provider.OPENROUTER, - maxTokens: 2048, - detail: 'high', - estimatedCostPerCall: 0.03, - resolutionReason: 'Multi-image comparison — Claude best at nuanced visual reasoning', - }; - }Also applies to: 266-267
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/vision/index.ts` around lines 192 - 205, The condition at line 192 that checks `complexity >= TaskComplexity.HIGH` returns early and prevents the multi-image competitor comparison branch at line 204 from executing. Move the imageCount > 1 check before the complexity-based early return at line 192, or add a condition to exclude multi-image scenarios from the early return in the condition. This ensures that competitor comparison tasks with multiple images reach the intended Claude model path in buildCompetitorComparisonTask instead of being short-circuited by the complexity check.packages/llm-router/src/providers/openrouter.ts-52-60 (1)
52-60:⚠️ Potential issue | 🟠 MajorAdd explicit timeout configuration to both provider clients to prevent extended blocking on stalled upstream connections. Both OpenAI SDK clients lack explicit timeout settings, relying on the SDK's 10-minute default, which is too long for consistent availability control in a request-handling service.
packages/llm-router/src/providers/openrouter.ts#L52-L60: Set explicit timeout (e.g.,timeout: 30000for 30 seconds) in the OpenAI client constructor.packages/llm-router/src/providers/perplexity.ts#L25-L29: Apply the same explicit timeout configuration for consistency.Consider using
AbortSignalfor finer control over the entire response lifecycle, as the SDK's timeout only protects the initial response headers, not the body streaming phase.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/src/providers/openrouter.ts` around lines 52 - 60, Both OpenAI SDK client initializations in the openrouter.ts (lines 52-60) and perplexity.ts (lines 25-29) files lack explicit timeout configuration, causing them to rely on the SDK's default 10-minute timeout which is too long for availability control. In packages/llm-router/src/providers/openrouter.ts, add a timeout property (e.g., timeout: 30000) to the OpenAI client constructor configuration object. Apply the same explicit timeout configuration to the OpenAI client constructor in packages/llm-router/src/providers/perplexity.ts for consistency. This ensures both provider clients have explicit timeout settings to prevent extended blocking on stalled upstream connections.
🟡 Minor comments (1)
src/services/llm.ts-75-88 (1)
75-88:⚠️ Potential issue | 🟡 Minor | ⚡ Quick win
generateContentexposescontextbut never uses it.Line 77 advertises
options.context, but Lines 79-88 ignore it, so callers cannot supply generation context despite the method contract.Suggested fix
async generateContent( prompt: string, options?: { complexity?: TaskComplexity; context?: string }, ): Promise<string> { + const userPrompt = options?.context + ? `${options.context}\n\n${prompt}` + : prompt; + const response = await this.router.execute( { clientId: this.clientId, type: TaskType.CONTENT_GENERATION, complexity: options?.complexity ?? TaskComplexity.MEDIUM, description: 'Website content generation', }, 'You are an expert website copywriter. Generate compelling, conversion-optimized content that is factual, professional, and aligned with the brand voice. Never invent credentials, certifications, or claims.', - prompt, + userPrompt, ); return response.content; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/services/llm.ts` around lines 75 - 88, The generateContent method in src/services/llm.ts accepts an options parameter with a context property but never uses it. To fix this, incorporate the context value from options?.context into the request sent to this.router.execute, either by appending it to the prompt string being passed as the third argument or by including it in the request object as a field that router.execute can utilize for enhanced content generation.
🧹 Nitpick comments (4)
packages/llm-router/README.md (2)
159-173: 💤 Low valueAdd language specifier to file structure diagram code block.
Line 159 opens a fenced code block for a directory tree diagram but does not specify a language. Add
```textto comply with markdown linting (MD040).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/README.md` around lines 159 - 173, The file structure diagram code block at line 159 in the README.md file is missing a language specifier on the opening backticks. Add the language specifier text to the code block opening by changing the opening ``` to ```text to comply with markdown linting rule MD040.Source: Linters/SAST tools
34-46: 💤 Low valueAdd language specifier to architecture diagram code block.
Line 34 opens a fenced code block for an ASCII architecture diagram but does not specify a language. Add
```textto comply with markdown linting (MD040).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/llm-router/README.md` around lines 34 - 46, The fenced code block in the README.md file that contains the ASCII architecture diagram is missing a language specifier. Change the opening fence from ``` to ```text at the start of the diagram (before the line showing "L9LLMRouter.execute()") to properly specify that it contains plain text content and comply with markdown linting rule MD040.Source: Linters/SAST tools
docs/recursive_optimization_report.md (1)
28-42: ⚡ Quick winFix table column count in File Structure Alignment section.
Lines 30–42 define a 3-column table (Check, Status, Evidence) in the header, but the data rows provide only 2 columns. The "Evidence" values are embedded within the Status column as markdown comments. Separate the Evidence values into their own column for proper table structure.
Example correction for lines 31–35:
| Check | Status | Evidence | |-------|--------|----------| | L9_META on tracked files | PARTIAL | new files (`src/services/llm.ts`, `scripts/verify-visual-qa.mjs`) lack headers | | No empty dirs | PASS | — | | Workspace package correctly placed | PASS | `packages/llm-router/` |🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/recursive_optimization_report.md` around lines 28 - 42, The File Structure Alignment section defines a 3-column table (Check, Status, Evidence) but the data rows only provide 2 columns with Evidence text embedded within the Status column. Separate each Evidence value into its own third column to match the table header structure. For each row in the File Structure Alignment table, move the evidence text that currently appears after the Status value (and any dashes) into a separate Evidence column cell. Apply the same fix to the Schema and Field Alignment section table which has the identical column count mismatch.Source: Linters/SAST tools
DEPLOYMENT.md (1)
120-127: 💤 Low valueRepetitive sentence structure in the "Do Not" list.
Lines 122–127 contain six successive sentences beginning with "Do not." This creates a monotonous reading experience. Consider rewording some clauses for variety while preserving the imperative tone.
Example refactor:
- Do not deploy production before preview passes. - Commit no `.env.local` file. - Hardcoding of API keys or Vercel tokens is prohibited. - Avoid calling deployment successful without URL and verification evidence. - Do not treat local build success as deployment proof. - The build:router step must run before build — the Astro site depends on the compiled router.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@DEPLOYMENT.md` around lines 120 - 127, The "Do Not" list in the DEPLOYMENT.md file contains six successive bullet points that all begin with "Do not," creating a repetitive and monotonous reading experience. Refactor the list items to introduce varied sentence structures while maintaining the imperative tone and the core safety messages. Reword at least half of the items using alternative phrasings such as passive voice constructions, prohibition phrases without "do not," or directive statements that convey the same requirement but with different grammatical patterns. Ensure all items remain clear, concise, and actionable directives for deployment best practices.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/llm-router/package.json`:
- Around line 5-7: The package.json for llm-router is missing the "type":
"module" declaration which is required for Node.js to correctly recognize and
parse the ESM modules being exported. Add "type": "module" as a top-level field
in the package.json file, placing it alongside other metadata fields like "main"
and "types". This ensures that Node.js will interpret the .js files in the
exports field as ES modules rather than CommonJS, preventing SyntaxError when
consumers import the package.
In `@packages/llm-router/src/index.ts`:
- Around line 117-121: The budget downgrade at lines 117-121 updates
decision.model but the execution paths at lines 127-128, 146-150, and 158-160
recompute configuration from the original task object, ignoring the downgraded
model. At each execution path location (lines 127-128, 146-150, and 158-160),
replace the configuration recomputation that uses task to instead use
decision.model (the potentially downgraded model value). This ensures the
throttling policy is actually applied at runtime by using the downgraded model
for all subsequent configuration decisions, not just storing it in the decision
object.
- Around line 129-137: When consensus mode is enabled and completeWithConsensus
is called in the block at lines 129-137, multiple provider calls are made to
achieve consensus. However, at line 169-173 where the spend is recorded to the
budget tracker, only the cost from result.best is captured, which undercounts
the actual total spend. Modify the cost tracking logic at lines 169-173 to
extract and record the total cost from the entire consensus result object (which
represents all provider calls made during consensus) rather than just the cost
from result.best, ensuring accurate budget enforcement that reflects actual
spending.
In `@scripts/verify-visual-qa.mjs`:
- Line 106: The dynamic import of createWebsiteFactoryLLM from
../src/services/llm.ts at line 106 will fail at runtime because plain Node.js
cannot resolve .ts files in ESM context without a TypeScript loader. Either
modify the import to use the compiled JavaScript version of the llm module (if
available in your build output), or change the script invocation from plain node
to use npx tsx which provides TypeScript support for ESM imports.
---
Major comments:
In @.env.example:
- Around line 6-7: The PERPLEXITY_API_KEY and OPENROUTER_API_KEY environment
variables in the .env.example file are populated with fake token-shaped
placeholder values that can falsely satisfy non-empty validation checks, making
the system appear ready when it lacks actual credentials. Replace these fake
values with empty strings to ensure that validation properly fails when real
operator-owned secrets are not configured, following the fail-closed validation
principle and avoiding the practice of inventing placeholder values.
In `@contracts/llm_router_integration.yaml`:
- Line 57: The provider field contains non-canonical provider identifiers that
mix provider names with model/modality metadata. At line 57, change `provider:
openrouter (vision model)` to use only the canonical provider name `openrouter`
in the provider field, and add a separate field (such as `modality` or
`variant`) to capture the vision model descriptor. Apply the same normalization
at line 66 where a similar mixed provider identifier appears. This ensures
contract consumers can parse the provider deterministically without string
parsing.
In `@DEPLOYMENT.md`:
- Around line 37-43: The environment variable names and launch-requirement scope
in DEPLOYMENT.md (lines 37–43) do not match the authoritative definitions in
config/launch-env.required.yaml. Fix this in two places: (1) In DEPLOYMENT.md
lines 37–43, correct the variable names: change PUBLIC_FORM_ENDPOINT to
FORM_ENDPOINT_URL and PUBLIC_ANALYTICS_ID to ANALYTICS_MEASUREMENT_ID; clarify
the section scope by either relabeling it to indicate conditional-feature
requirements (not universal launch requirements) or remove ACCULYNX_API_KEY from
the "required for launch" section since config/launch-env.required.yaml marks
the acculynx group as required_for_launch: false. (2) In
docs/recursive_optimization_report.md lines 84–88, update the V003 FIXED
statement to reflect the actual corrective action taken (the name corrections
and scope clarification in DEPLOYMENT.md).
In `@Makefile`:
- Around line 91-92: The deploy-production target in the Makefile currently
executes directly without enforcing required quality gates. Add verify:visual-qa
(and launch-env as mentioned in the requirement title) as prerequisites to the
deploy-production target to enforce the preview-first/evidence-backed release
control path. In Makefile syntax, list these prerequisites on the target
declaration line before the colon, separated by spaces, so they must complete
successfully before the npm run deploy:production command executes.
In `@packages/llm-router/src/budget/index.ts`:
- Around line 86-94: The recordSpend method does not validate the amount
parameter before updating budget state, allowing negative or non-finite values
to corrupt the budget tracking. Add a validation guard at the beginning of the
recordSpend method that rejects negative amounts and non-finite values (NaN,
Infinity) by throwing an error before any state mutations occur, ensuring only
valid spend amounts are recorded.
- Around line 146-149: The evaluateTask method receives an estimatedCost
parameter that is currently unused when computing throttle decisions. To fix
this, modify the method to compute throttle based on projected spend rather than
current spend only. After calling getState to retrieve the current state, create
a projected state by adding the estimatedCost to the current spend amounts, then
pass this projected state to computeThrottleLevel instead of passing the current
state. This ensures that throttle decisions account for the immediate impact of
the task being evaluated.
In `@packages/llm-router/src/index.ts`:
- Around line 145-157: Vision tasks are silently falling through to the text
generation branch when images are not provided, instead of failing with a clear
error. Add a validation check that detects when a task type is in
VISION_TASK_TYPES but options.images is missing or empty, and throw an error
with a descriptive message about the missing images before the condition on Line
145. This ensures vision task inputs are validated against their contract
requirements and fail fast rather than producing incorrect results.
In `@packages/llm-router/src/matrices/perplexity-matrix.ts`:
- Around line 168-170: The TaskComplexity string enum has lexical ordering that
contradicts semantic ordering (alphabetically: critical < high < low < medium <
trivial, but semantically: trivial < low < medium < high < critical), causing
all relational comparisons to produce incorrect results. Create an explicit
numeric rank map for TaskComplexity (mapping TRIVIAL=0, LOW=1, MEDIUM=2, HIGH=3,
CRITICAL=4), then use rank comparisons instead of direct enum comparisons. In
packages/llm-router/src/matrices/perplexity-matrix.ts lines 168-170, replace the
relational comparison with a rank-based comparison for the context size
determination. In packages/llm-router/src/matrices/general-matrix.ts lines
213-219, replace the relational comparisons used for token allocation with
rank-based comparisons. Additionally, apply the same rank-based comparison fix
to the remaining TaskComplexity comparisons in
packages/llm-router/src/matrices/perplexity-matrix.ts at lines 180, 226, 249,
and 290.
In `@packages/llm-router/src/providers/openrouter.ts`:
- Around line 52-60: Both OpenAI SDK client initializations in the openrouter.ts
(lines 52-60) and perplexity.ts (lines 25-29) files lack explicit timeout
configuration, causing them to rely on the SDK's default 10-minute timeout which
is too long for availability control. In
packages/llm-router/src/providers/openrouter.ts, add a timeout property (e.g.,
timeout: 30000) to the OpenAI client constructor configuration object. Apply the
same explicit timeout configuration to the OpenAI client constructor in
packages/llm-router/src/providers/perplexity.ts for consistency. This ensures
both provider clients have explicit timeout settings to prevent extended
blocking on stalled upstream connections.
In `@packages/llm-router/src/providers/perplexity.ts`:
- Around line 65-68: The disableSearch configuration option is ineffective
because web_search_options is unconditionally set at lines 60-63 in
packages/llm-router/src/providers/perplexity.ts while the check at lines 65-68
applies no behavior. To fix this, move the web_search_options assignment at
lines 60-63 inside the conditional block so it only gets set when
config.disableSearch is false (meaning search is enabled). This ensures that
when disableSearch is true, web_search_options is not applied and search remains
disabled as intended. Remove or restructure the empty conditional at lines 65-68
after moving the logic.
In `@packages/llm-router/src/types.ts`:
- Around line 16-17: Remove the OPENAI_DIRECT and ANTHROPIC_DIRECT enum values
from the Provider enum in the types.ts file. These providers are not part of the
locked integration (only OpenRouter and Perplexity are supported at runtime),
and keeping them in the public contract exposes invalid routing states. Retain
only the Provider enum values that correspond to the actually supported runtime
providers.
- Around line 50-53: The vision model enum members (GPT4O_VISION,
CLAUDE_SONNET_VISION, GEMINI_FLASH_VISION) in
packages/llm-router/src/types.ts#L50-L53 are assigned identical string values to
their base model counterparts, causing object key collisions when used as Record
keys. Assign unique string values to each vision variant by appending a suffix
(e.g., change 'openai/gpt-4o' to 'openai/gpt-4o-vision' for GPT4O_VISION,
'anthropic/claude-sonnet-4' to 'anthropic/claude-sonnet-4-vision' for
CLAUDE_SONNET_VISION, and 'google/gemini-2.5-flash' to
'google/gemini-2.5-flash-vision' for GEMINI_FLASH_VISION). After fixing the enum
values, verify that the Record<GeneralModel, ...> definitions in
packages/llm-router/src/matrices/general-matrix.ts#L183-L199 (fallback chains,
cost maps, and provider model IDs) work correctly with the now-unique values; no
direct changes should be needed at this location once the enum is corrected.
In `@packages/llm-router/src/vision/index.ts`:
- Around line 316-336: The task builders buildLayoutValidationTask,
buildCompetitorComparisonTask, and buildConversionAuditTask expect screenshot
image URLs as their first parameter (defined as screenshotUrl at lines 242-243,
258-259, and 274-275 respectively), but in the generateFullSiteQAPlan function,
webpage URLs from config.pages and config.competitorUrl are being passed
directly to these builders at lines 318, 326, and 335. Convert the webpage URLs
to screenshot image URLs before passing them to each task builder call. Ensure
that wherever page URLs are forwarded to these three task builders, they are
first transformed into the corresponding screenshot URLs that the builders
expect.
- Around line 192-205: The condition at line 192 that checks `complexity >=
TaskComplexity.HIGH` returns early and prevents the multi-image competitor
comparison branch at line 204 from executing. Move the imageCount > 1 check
before the complexity-based early return at line 192, or add a condition to
exclude multi-image scenarios from the early return in the condition. This
ensures that competitor comparison tasks with multiple images reach the intended
Claude model path in buildCompetitorComparisonTask instead of being
short-circuited by the complexity check.
In `@scripts/verify-visual-qa.mjs`:
- Around line 133-135: The else block at lines 133-135 only logs a message when
AI analysis is skipped due to missing OPENROUTER_API_KEY, but does not persist
the visual_qa_report.json file. This means no artifact is created to provide
evidence of the verification. Modify the else branch to emit or write the
visual_qa_report.json file (containing the screenshots data captured before this
point) to disk, ensuring that verification evidence is always persisted
regardless of whether AI analysis was performed or skipped.
- Around line 45-48: The preflight check at lines 45-48 only validates
OPENROUTER_API_KEY, but the createWebsiteFactoryLLM() call at lines 103-107
requires both OPENROUTER_API_KEY and another provider key to be set. Update the
guard condition at lines 45-48 to check for both keys that
createWebsiteFactoryLLM() requires, so that missing dependencies are caught
early with a clear message rather than causing a runtime failure when the
factory is instantiated at lines 103-107. This ensures the preflight validation
matches the actual contract of createWebsiteFactoryLLM().
- Line 180: The catch handler for the main() promise at line 180
(main().catch(console.error)) only logs errors to console but does not exit the
process with a failure status. Modify the error handler to call process.exit(1)
after logging the error, ensuring that the process terminates with a non-zero
exit code when main() rejects, which will properly signal verification failures
to CI systems.
- Line 123: The status check on line 123 uses case-sensitive string matching
with includes('critical'), which fails to catch variations like 'Critical' or
'CRITICAL', leading to false PASS outcomes. Fix this by converting the analysis
string to lowercase before checking if it includes the critical keyword,
ensuring the FAIL/PASS gate properly catches all case variations of critical
severity levels.
- Line 63: The execSync call at line 63 interpolates siteUrl directly into a
shell command string, creating a command-injection vulnerability, and does not
verify that the HTTP status code indicates success. Replace the string
interpolation with argument-safe process execution by passing the URL as a
separate argument to curl, and explicitly check that the returned HTTP status
code falls within the 2xx success range before proceeding. If the status code
indicates failure (4xx, 5xx, etc.), throw an error to halt execution.
In `@src/services/llm.ts`:
- Around line 286-289: The configuration fallback on lines 286-288 is fail-open:
using a shared 'default' clientId and silently coercing invalid numeric
environment values. Replace these lenient defaults with strict validation that
throws an error if required values are missing or invalid. Remove the fallback
to 'default' for clientId and instead require it to be explicitly provided;
validate that MONTHLY_BUDGET_PER_CLIENT and WEEKLY_BUDGET_TARGET are valid
numbers, throwing an error if they are missing or cannot be parsed. This
validation should be enforced through the launch environment check process to
catch misconfigurations during deployment rather than silently using defaults
that could collapse tenant isolation.
---
Minor comments:
In `@src/services/llm.ts`:
- Around line 75-88: The generateContent method in src/services/llm.ts accepts
an options parameter with a context property but never uses it. To fix this,
incorporate the context value from options?.context into the request sent to
this.router.execute, either by appending it to the prompt string being passed as
the third argument or by including it in the request object as a field that
router.execute can utilize for enhanced content generation.
---
Nitpick comments:
In `@DEPLOYMENT.md`:
- Around line 120-127: The "Do Not" list in the DEPLOYMENT.md file contains six
successive bullet points that all begin with "Do not," creating a repetitive and
monotonous reading experience. Refactor the list items to introduce varied
sentence structures while maintaining the imperative tone and the core safety
messages. Reword at least half of the items using alternative phrasings such as
passive voice constructions, prohibition phrases without "do not," or directive
statements that convey the same requirement but with different grammatical
patterns. Ensure all items remain clear, concise, and actionable directives for
deployment best practices.
In `@docs/recursive_optimization_report.md`:
- Around line 28-42: The File Structure Alignment section defines a 3-column
table (Check, Status, Evidence) but the data rows only provide 2 columns with
Evidence text embedded within the Status column. Separate each Evidence value
into its own third column to match the table header structure. For each row in
the File Structure Alignment table, move the evidence text that currently
appears after the Status value (and any dashes) into a separate Evidence column
cell. Apply the same fix to the Schema and Field Alignment section table which
has the identical column count mismatch.
In `@packages/llm-router/README.md`:
- Around line 159-173: The file structure diagram code block at line 159 in the
README.md file is missing a language specifier on the opening backticks. Add the
language specifier text to the code block opening by changing the opening ``` to
```text to comply with markdown linting rule MD040.
- Around line 34-46: The fenced code block in the README.md file that contains
the ASCII architecture diagram is missing a language specifier. Change the
opening fence from ``` to ```text at the start of the diagram (before the line
showing "L9LLMRouter.execute()") to properly specify that it contains plain text
content and comply with markdown linting rule MD040.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a4946c3c-e193-4f6f-88bd-d599a6644ab6
📒 Files selected for processing (25)
.env.exampleAGENTS.mdARCHITECTURE.mdDEPLOYMENT.mdMakefileREADME.mdVALIDATION.mdconfig/launch-env.required.yamlcontracts/llm_router_integration.yamldocs/recursive_optimization_report.mdpackage.jsonpackages/llm-router/README.mdpackages/llm-router/package.jsonpackages/llm-router/src/budget/index.tspackages/llm-router/src/index.tspackages/llm-router/src/matrices/general-matrix.tspackages/llm-router/src/matrices/perplexity-matrix.tspackages/llm-router/src/providers/openrouter.tspackages/llm-router/src/providers/perplexity.tspackages/llm-router/src/types.tspackages/llm-router/src/vision/index.tspackages/llm-router/tsconfig.jsonscripts/verify-visual-qa.mjssrc/services/llm.tstsconfig.json
| "main": "dist/index.js", | ||
| "types": "dist/index.d.ts", | ||
| "scripts": { |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify package module-mode alignment (read-only)
jq -r '.type // "<missing>"' packages/llm-router/package.json
jq -r '.compilerOptions.module' packages/llm-router/tsconfig.json
rg -n '^\s*export\s+|^\s*import\s+' packages/llm-router/src -g '*.ts'Repository: cryptoxdog/Website-Bot
Length of output: 5788
🏁 Script executed:
cat -n packages/llm-router/package.jsonRepository: cryptoxdog/Website-Bot
Length of output: 1807
Add "type": "module" to properly declare this package as ESM.
The package exports ESM modules via the "exports" field (lines 32–49) with "import" conditions pointing to .js files, but omits "type": "module". TypeScript is configured to emit ESNext format, producing ES module syntax in the compiled .js files. Without "type": "module", Node.js will attempt to parse these files as CommonJS, causing SyntaxError when consumers import the package.
Proposed fix
{
"name": "`@l9/llm-router`",
"version": "1.0.0",
+ "type": "module",
"description": "Reusable multi-provider LLM routing module with task-to-model matrix, Perplexity search depth optimization, vision QA, and budget enforcement. Shared across all L9 bots.",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llm-router/package.json` around lines 5 - 7, The package.json for
llm-router is missing the "type": "module" declaration which is required for
Node.js to correctly recognize and parse the ESM modules being exported. Add
"type": "module" as a top-level field in the package.json file, placing it
alongside other metadata fields like "main" and "types". This ensures that
Node.js will interpret the .js files in the exports field as ES modules rather
than CommonJS, preventing SyntaxError when consumers import the package.
| if (throttle.forceDowngrade) { | ||
| decision.downgraded = true; | ||
| decision.downgradedFrom = decision.model; | ||
| decision.model = this.getDowngradedModel(decision.model, throttle.maxModelTier); | ||
| } |
There was a problem hiding this comment.
Budget downgrade mutates decision only, not execution config.
Lines 117-121 update decision.model, but execution paths recompute configs from task at Lines 127/146/158. The downgraded model is therefore not actually used, so throttling policy is bypassed at runtime.
Also applies to: 127-128, 146-150, 158-160
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llm-router/src/index.ts` around lines 117 - 121, The budget
downgrade at lines 117-121 updates decision.model but the execution paths at
lines 127-128, 146-150, and 158-160 recompute configuration from the original
task object, ignoring the downgraded model. At each execution path location
(lines 127-128, 146-150, and 158-160), replace the configuration recomputation
that uses task to instead use decision.model (the potentially downgraded model
value). This ensures the throttling policy is actually applied at runtime by
using the downgraded model for all subsequent configuration decisions, not just
storing it in the decision object.
| if (options?.consensus && config.variations > 1) { | ||
| const result = await this.perplexity.completeWithConsensus( | ||
| config, | ||
| systemPrompt, | ||
| userPrompt, | ||
| options.assistantContext, | ||
| ); | ||
| response = result.best; | ||
| } else { |
There was a problem hiding this comment.
Consensus mode under-reports spend to the budget tracker.
At Line 130, consensus triggers multiple provider calls, but Line 169 records only response.cost from result.best. This undercounts real spend and weakens budget enforcement.
Suggested fix
- let response: LLMResponse;
+ let response: LLMResponse;
+ let billedCost = 0;
...
if (options?.consensus && config.variations > 1) {
const result = await this.perplexity.completeWithConsensus(
config,
systemPrompt,
userPrompt,
options.assistantContext,
);
response = result.best;
+ billedCost = result.all.reduce((sum, r) => sum + r.cost, 0);
} else {
response = await this.perplexity.complete(
config,
systemPrompt,
userPrompt,
options?.assistantContext,
);
+ billedCost = response.cost;
}
...
- this.budget.recordSpend(task.clientId, response.cost);
+ this.budget.recordSpend(task.clientId, billedCost || response.cost);
...
- decision.actualCost = response.cost;
+ decision.actualCost = billedCost || response.cost;Also applies to: 169-173
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llm-router/src/index.ts` around lines 129 - 137, When consensus mode
is enabled and completeWithConsensus is called in the block at lines 129-137,
multiple provider calls are made to achieve consensus. However, at line 169-173
where the spend is recorded to the budget tracker, only the cost from
result.best is captured, which undercounts the actual total spend. Modify the
cost tracking logic at lines 169-173 to extract and record the total cost from
the entire consensus result object (which represents all provider calls made
during consensus) rather than just the cost from result.best, ensuring accurate
budget enforcement that reflects actual spending.
| if (process.env.OPENROUTER_API_KEY) { | ||
| console.log('\n🤖 Running vision analysis...\n'); | ||
| // Dynamic import of the LLM service | ||
| const { createWebsiteFactoryLLM } = await import('../src/services/llm.ts'); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== verify-visual-qa invocation =="
rg -n "verify-visual-qa|scripts/verify-visual-qa\\.mjs" package.json Makefile || true
echo
echo "== TS loaders/runtime wrappers =="
rg -n "tsx|ts-node|--loader|NODE_OPTIONS" package.json Makefile || trueRepository: cryptoxdog/Website-Bot
Length of output: 886
🏁 Script executed:
cat -n scripts/verify-visual-qa.mjs | head -110 | tail -20Repository: cryptoxdog/Website-Bot
Length of output: 901
🏁 Script executed:
#!/bin/bash
# Check for NODE_OPTIONS or loader setup in package.json scripts
rg -A2 'verify:visual-qa|verify-visual-qa' package.jsonRepository: cryptoxdog/Website-Bot
Length of output: 260
🏁 Script executed:
# Check if there's a build step or transpilation happening
rg -n "build|compile|tsx|ts-node" package.json | grep -E "scripts|dependencies"Repository: cryptoxdog/Website-Bot
Length of output: 122
🏁 Script executed:
# Look for .js version of the llm.ts file (in case it's pre-built)
find . -name "llm.*" -type f 2>/dev/null | grep -E "(src|dist|build)" || trueRepository: cryptoxdog/Website-Bot
Length of output: 88
🏁 Script executed:
# Check for any build/dist output or tsconfig configuration
rg -n "outDir|rootDir|declaration" tsconfig.json || echo "No tsconfig.json found"Repository: cryptoxdog/Website-Bot
Length of output: 144
🏁 Script executed:
# Look for any build scripts or preprocessing
rg -n "build|dist|prepare|postinstall" package.json | head -20Repository: cryptoxdog/Website-Bot
Length of output: 254
🏁 Script executed:
# Check if there's a src/services/llm.js that gets built
ls -la src/services/ 2>/dev/null || echo "src/services not found"Repository: cryptoxdog/Website-Bot
Length of output: 237
🏁 Script executed:
# Check if Astro or any other tool might be handling the TS import
cat -n tsconfig.jsonRepository: cryptoxdog/Website-Bot
Length of output: 725
🏁 Script executed:
# Verify if there's a .mjs config or any loader setup
cat -n package.json | grep -A10 -B10 "verify:visual-qa"Repository: cryptoxdog/Website-Bot
Length of output: 1187
🏁 Script executed:
# Check if the script actually runs or if there are any comments about it
head -30 scripts/verify-visual-qa.mjsRepository: cryptoxdog/Website-Bot
Length of output: 1085
Line 106 will fail at runtime with plain Node.js execution—.ts imports require explicit loader configuration.
The script is invoked as node scripts/verify-visual-qa.mjs with no TypeScript loader. When OPENROUTER_API_KEY is set, the dynamic import at line 106 will fail because Node.js cannot natively resolve .ts files in ESM context. Only the builder scripts use npx tsx to handle TypeScript. Either switch this script to use npx tsx or generate a .js output of llm.ts as part of the build.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/verify-visual-qa.mjs` at line 106, The dynamic import of
createWebsiteFactoryLLM from ../src/services/llm.ts at line 106 will fail at
runtime because plain Node.js cannot resolve .ts files in ESM context without a
TypeScript loader. Either modify the import to use the compiled JavaScript
version of the llm module (if available in your build output), or change the
script invocation from plain node to use npx tsx which provides TypeScript
support for ESM imports.
1. packages/llm-router/package.json: Add "type": "module" for ESM declaration 2. packages/llm-router/src/index.ts: Add clientId ?? 'default' fallback in route() (TaskDescriptor.clientId is optional, RoutingDecision.clientId is required) 3. src/services/llm.ts: Fix budget property names to match BudgetConfig interface - weeklyCeiling → weeklyHardCeiling - globalHardCeiling → globalMonthlyHardCeiling - surgeAllowance: true → surgeThreshold: 0.6 4. src/services/llm.ts: Fix planFullSiteQA to match FullSiteQAConfig interface - Remove non-existent siteUrl, includeCompetitorComparison, includeConversionAudit - Use correct: pages, viewports, competitorUrl?, conversionAudit 5. packages/llm-router/src/index.ts: Propagate budget downgrade to execution config (previously decision.model was set but execution paths recomputed from task) 6. packages/llm-router/src/index.ts: Track total consensus cost across all calls (previously only recorded result.best.cost, missing other variation costs) 7. scripts/verify-visual-qa.mjs: Fix .ts import → .js (no TS loader in .mjs runtime)
…ter adapter
Website-Bot now executes all LLM calls through the shared router package, so
model selection / budgeting / provider management live in one place (the router)
and are changed by bumping a dependency rather than editing per-repo code.
- src/services/llm.ts: rewritten over L9LLMRouter. createWebsiteFactoryLLM keeps
its exact public interface (generateContent/designReasoning/generateSchema/
recordUsage/flushUsage) so the pipeline stages are unchanged. Maps the three
task methods onto TaskDescriptors (CONTENT_GENERATION/MEDIUM,
STRATEGIC_REASONING/HIGH, CODE_GENERATION/LOW) per
contracts/llm_router_integration.yaml. Router is constructed lazily on first
use, so a dry run stays side-effect-free and needs no OPENROUTER_API_KEY.
Errors (incl. BudgetExhaustedError) are surfaced as BuildError('LLM_CALL_FAILED').
- package.json: add "@quantum-l9/llm-router": "^1.0.0".
- Delete vendored packages/llm-router/ (now sourced from the registry); drop the
dependabot entry and the Makefile build-router target.
- Reconcile docs/contract (README, AGENTS, ARCHITECTURE, DEPLOYMENT, PR template,
llm_router_integration.yaml) to describe the router as a GitHub Packages
dependency rather than a vendored workspace.
NOTE: install / typecheck / CI stay red until @quantum-l9/llm-router@1.0.0 is
published and granted read access (LLM-Router PR #1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqWfuXWTn5jo8f5fnkSozx
2 participants
Summary
Wires the
@l9/llm-routershared module into the Website Factory Bot as a workspace dependency. This gives the Website Bot the same intelligent multi-provider model routing, budget enforcement, and visual QA capabilities as the SEO Bot.Changes
New Files (14)
packages/llm-router/— Full @l9/llm-router source (workspace package)src/services/llm.ts— Website Factory LLM service wrapperscripts/verify-visual-qa.mjs— Visual QA verification scriptcontracts/llm_router_integration.yaml— Integration contract.env.example— Environment variable templatetsconfig.json— TypeScript configurationdocs/recursive_optimization_report.md— Optimization audit trailModified Files (8)
Makefile— 5 new targets addedpackage.json— Workspace config + new scriptsconfig/launch-env.required.yaml— Added llm_intelligence groupDEPLOYMENT.md— Rewritten for current env surfaceVALIDATION.md— Added Launch Env + Visual QA classesREADME.md— Updated quick start and project identityAGENTS.md— LLM Router as locked decisionARCHITECTURE.md— Router layer documentedRecursive Optimization
Ran
/l9-recursive-optimization(optimize mode, persist=apply):Integration Architecture
Budget
Visual QA
validation/visual_qa_report.jsonSummary by CodeRabbit
Release Notes
New Features
Enhancements