-
Notifications
You must be signed in to change notification settings - Fork 0
Phase 1 Wave 1 — LLMProvider seam types (1.A) + ToolNormalizer (1.E) + CostTracker (1.B) #7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
2024549
docs(roadmap): mark 1.L.0 done (PR #6 merged); open Wave 1 (1.A ‖ 1.L)
cemililik a5aa707
feat(llm): freeze the LLMProvider seam types (1.A)
cemililik 7ca81b3
feat(llm): CostTracker + model pricing (1.B) and ToolNormalizer (1.E)
cemililik db9206c
fix(llm): tighten seam tool-schema validation and gemini reshape
cemililik ef4eee7
docs(repo): mark Phase 1 Wave 1 (1.A/1.B/1.E) in review under PR #7
cemililik 328f32a
refactor(llm): cut reshapeNode complexity; clear sonar smells
cemililik 52e2b8e
fix(llm): harden ToolNormalizer + close PR #7 review findings
cemililik 8473263
test(llm): clear SonarCloud smells on the new PR #7 test code
cemililik 2a2cc42
test(llm): drop the `as JSONSchema7` test casts for runtime guards
cemililik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,21 @@ | ||
| # `@relavium/llm` — placeholder | ||
| # @relavium/llm | ||
|
|
||
| Relavium's own multi-LLM abstraction: the `LLMProvider` seam + thin hand-rolled | ||
| adapters over each provider's official SDK. **No vendor SDK type crosses the seam** | ||
| ([ADR-0011](../../docs/decisions/0011-internal-llm-abstraction.md)). | ||
| Relavium's **own** multi-LLM abstraction: the provider-agnostic **`LLMProvider` seam** plus | ||
| (later) thin hand-rolled adapters over each provider's official TypeScript SDK. No Vercel AI SDK, | ||
| no LangChain — see [ADR-0011](../../docs/decisions/0011-internal-llm-abstraction.md). | ||
|
|
||
| > **Not built yet.** This package is a directory placeholder. It is built out in | ||
| > [Phase 1 — engine and LLM](../../docs/roadmap/phases/phase-1-engine-and-llm.md). It has | ||
| > no `package.json` yet, so pnpm/Turborepo do not treat it as a workspace. | ||
| ## The immovable contract | ||
|
|
||
| Canonical contract: [llm-provider-seam.md](../../docs/reference/shared-core/llm-provider-seam.md). | ||
| The seam (`src/types.ts`) is expressed **only** in Relavium/Zod types. **No vendor SDK type ever | ||
| crosses it** — provider SDKs are imported _only_ inside `src/adapters/*` (enforced by the | ||
| import-zone lint fence). The canonical contract is | ||
| [docs/reference/shared-core/llm-provider-seam.md](../../docs/reference/shared-core/llm-provider-seam.md); | ||
| this package implements it. The seam's _shape_ is frozen; the adapters behind it are reversible, | ||
| and the _set_ of provider ids is meant to grow (an additive amendment, not a contract change). | ||
|
|
||
| ## Status | ||
|
|
||
| **1.A — seam types frozen.** The adapters (Anthropic, the OpenAI-compatible adapter serving | ||
| OpenAI + DeepSeek, Gemini), the `FallbackChain` runner, the `CostTracker`, and the conformance | ||
| suite land in workstreams 1.B–1.K (see | ||
| [docs/roadmap/phases/phase-1-engine-and-llm.md](../../docs/roadmap/phases/phase-1-engine-and-llm.md)). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| { | ||
| "name": "@relavium/llm", | ||
| "version": "0.0.0", | ||
| "private": true, | ||
| "type": "module", | ||
| "description": "Relavium's provider-agnostic LLMProvider seam + (later) thin hand-rolled adapters over the official provider SDKs (ADR-0011). No vendor SDK type crosses the seam.", | ||
| "exports": { | ||
| ".": { | ||
| "types": "./dist/index.d.ts", | ||
| "import": "./dist/index.js", | ||
| "default": "./dist/index.js" | ||
| } | ||
| }, | ||
| "main": "./dist/index.js", | ||
| "types": "./dist/index.d.ts", | ||
| "files": [ | ||
| "dist" | ||
| ], | ||
| "scripts": { | ||
| "build": "tsc -p tsconfig.build.json", | ||
| "typecheck": "tsc -p tsconfig.json --noEmit", | ||
| "lint": "eslint src", | ||
| "test": "vitest run" | ||
| }, | ||
| "dependencies": { | ||
| "@relavium/shared": "workspace:*", | ||
| "zod": "catalog:" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/json-schema": "catalog:", | ||
| "eslint": "catalog:", | ||
| "typescript": "catalog:", | ||
| "vitest": "catalog:" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| import { describe, expect, it } from 'vitest'; | ||
|
|
||
| import { CostTracker, cost, priceModel } from './cost-tracker.js'; | ||
| import { UnknownModelError } from './errors.js'; | ||
| import { KNOWN_MODEL_IDS, MODEL_PRICING, type ModelPricing } from './pricing.js'; | ||
|
|
||
| describe('priceModel', () => { | ||
| it('returns pricing for a known canonical model id', () => { | ||
| const p = priceModel('claude-opus-4-8'); | ||
| expect(p.provider).toBe('anthropic'); | ||
| expect(p.inputPerMtokMicrocents).toBe(500_000_000); // $5/MTok → 5e8 micro-cents | ||
| }); | ||
|
|
||
| it('throws a typed UnknownModelError (never a silent zero)', () => { | ||
| expect(() => priceModel('gpt-9-ultra')).toThrowError(UnknownModelError); | ||
| try { | ||
| priceModel('gpt-9-ultra'); | ||
| } catch (err) { | ||
| expect(err).toBeInstanceOf(UnknownModelError); | ||
| if (err instanceof UnknownModelError) { | ||
| expect(err.code).toBe('unknown_model'); | ||
| expect(err.modelId).toBe('gpt-9-ultra'); | ||
| expect(err.knownModels).toEqual(KNOWN_MODEL_IDS); | ||
| } | ||
| } | ||
| }); | ||
| }); | ||
|
|
||
| describe('cost', () => { | ||
| it('prices input + output in integer micro-cents (Opus 4.8)', () => { | ||
| // 1000 in @ $5/MTok = 500_000µ¢; 500 out @ $25/MTok = 1_250_000µ¢ | ||
| expect(cost('claude-opus-4-8', { inputTokens: 1000, outputTokens: 500 })).toBe(1_750_000); | ||
| }); | ||
|
|
||
| it('prices each token class once, including cache read + write (Opus 4.8)', () => { | ||
| expect( | ||
| cost('claude-opus-4-8', { | ||
| inputTokens: 1000, // 500_000 | ||
| outputTokens: 500, // 1_250_000 | ||
| cacheReadTokens: 2000, // 2000 @ $0.50/MTok = 100_000 | ||
| cacheWriteTokens: 4000, // 4000 @ $6.25/MTok = 2_500_000 | ||
| }), | ||
| ).toBe(4_350_000); | ||
| }); | ||
|
|
||
| it('prices Sonnet 4.6 and DeepSeek from their own rows', () => { | ||
| expect(cost('claude-sonnet-4-6', { inputTokens: 1000, outputTokens: 500 })).toBe(1_050_000); | ||
| // deepseek-chat: 1000 in @ $0.27/MTok = 27_000µ¢; 500 out @ $1.10/MTok = 55_000µ¢ | ||
| expect(cost('deepseek-chat', { inputTokens: 1000, outputTokens: 500 })).toBe(82_000); | ||
| }); | ||
|
|
||
| it('ignores cache-write tokens for a provider with no cache-write price (gpt-4o)', () => { | ||
| // gpt-4o has no cacheWritePerMtokMicrocents → the 1000 cache-write tokens cost 0. | ||
| expect(cost('gpt-4o', { inputTokens: 1000, outputTokens: 0, cacheWriteTokens: 1000 })).toBe( | ||
| 250_000, | ||
| ); // 1000 in @ $2.50/MTok = 250_000 | ||
| }); | ||
|
|
||
| it('rounds the per-class micro-cent figure (half-up)', () => { | ||
| // gpt-4o-mini cached input $0.075/MTok = 7_500_000µ¢/MTok → 1 token = 7.5 → rounds to 8. | ||
| expect(cost('gpt-4o-mini', { inputTokens: 0, outputTokens: 0, cacheReadTokens: 1 })).toBe(8); | ||
| }); | ||
| }); | ||
|
|
||
| describe('CostTracker', () => { | ||
| it('accumulates per-attempt cost across a failover', () => { | ||
| const tracker = new CostTracker(); | ||
| const a = tracker.record('claude-opus-4-8', { inputTokens: 1000, outputTokens: 500 }); | ||
| expect(a).toEqual({ | ||
| inputTokens: 1000, | ||
| outputTokens: 500, | ||
| costMicrocents: 1_750_000, | ||
| cumulativeCostMicrocents: 1_750_000, | ||
| }); | ||
| const b = tracker.record('gpt-4o', { inputTokens: 1000, outputTokens: 0 }); | ||
| expect(b.costMicrocents).toBe(250_000); | ||
| expect(b.cumulativeCostMicrocents).toBe(2_000_000); // 1_750_000 + 250_000 | ||
| expect(tracker.cumulativeCostMicrocents).toBe(2_000_000); | ||
| }); | ||
| }); | ||
|
|
||
| describe('MODEL_PRICING table invariants (the values seeded into model_catalog)', () => { | ||
| it('keys match KNOWN_MODEL_IDS and every catalog-projection field is complete + integer', () => { | ||
| const byLocale = (a: string, b: string): number => a.localeCompare(b); | ||
| expect(Object.keys(MODEL_PRICING).sort(byLocale)).toEqual([...KNOWN_MODEL_IDS].sort(byLocale)); | ||
| const rows: Array<[string, ModelPricing]> = Object.entries(MODEL_PRICING); | ||
| for (const [id, row] of rows) { | ||
| expect(row.nativeId.length, id).toBeGreaterThan(0); | ||
| expect(row.displayName.length, id).toBeGreaterThan(0); | ||
| expect(row.contextWindowTokens, id).toBeGreaterThan(0); | ||
| expect(row.maxOutputTokens, id).toBeGreaterThan(0); | ||
| // Prices are integer micro-cents per MTok — never a float, and never negative. | ||
| for (const price of [ | ||
| row.inputPerMtokMicrocents, | ||
| row.outputPerMtokMicrocents, | ||
| row.cachedInputPerMtokMicrocents, | ||
| row.cacheWritePerMtokMicrocents ?? 0, | ||
| ]) { | ||
| expect(Number.isInteger(price), id).toBe(true); | ||
| expect(price, id).toBeGreaterThanOrEqual(0); | ||
| } | ||
| } | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import { UnknownModelError } from './errors.js'; | ||
| import { | ||
| isCanonicalModelId, | ||
| KNOWN_MODEL_IDS, | ||
| MODEL_PRICING, | ||
| type ModelPricing, | ||
| } from './pricing.js'; | ||
| import type { Usage } from './types.js'; | ||
|
|
||
| /** | ||
| * Cost tracking Relavium owns, keyed on the canonical model id — never read from a provider | ||
| * field (1.B). All figures are integer **micro-cents** (1 micro-cent = 1e-8 USD), consistent with | ||
| * the seam's `Usage.costMicrocents` and the `cost:updated` event. | ||
| */ | ||
|
|
||
| /** Look up pricing for a canonical model id; throws `UnknownModelError` (never a silent zero). */ | ||
| export function priceModel(modelId: string): ModelPricing { | ||
| if (!isCanonicalModelId(modelId)) { | ||
| throw new UnknownModelError(modelId, KNOWN_MODEL_IDS); | ||
| } | ||
| return MODEL_PRICING[modelId]; | ||
| } | ||
|
|
||
| const TOKENS_PER_MTOK = 1_000_000; | ||
|
|
||
| /** | ||
| * The integer micro-cent cost of one usage record. Each token class is `tokens × | ||
| * pricePerMtokMicrocents / 1e6`, rounded once (the only float in the path; everything downstream | ||
| * sums integers). | ||
| * | ||
| * **Convention:** `inputTokens` is the count billed at the full input rate — cache-read and | ||
| * cache-write tokens are **separate** counts, not included in `inputTokens`. Adapters normalize to | ||
| * this (Anthropic's `input_tokens` is already net; the OpenAI-compatible adapter subtracts the | ||
| * cached subset). So each token class is billed exactly once. | ||
| */ | ||
| export function cost(modelId: string, usage: Usage): number { | ||
| const p = priceModel(modelId); | ||
| const cacheReadTokens = usage.cacheReadTokens ?? 0; | ||
| const cacheWriteTokens = usage.cacheWriteTokens ?? 0; | ||
| const perClass = (tokens: number, ratePerMtok: number): number => | ||
| Math.round((tokens * ratePerMtok) / TOKENS_PER_MTOK); | ||
| return ( | ||
| perClass(usage.inputTokens, p.inputPerMtokMicrocents) + | ||
| perClass(usage.outputTokens, p.outputPerMtokMicrocents) + | ||
| perClass(cacheReadTokens, p.cachedInputPerMtokMicrocents) + | ||
| perClass(cacheWriteTokens, p.cacheWritePerMtokMicrocents ?? 0) | ||
| ); | ||
| } | ||
|
|
||
| /** The cost figures for one `cost:updated` event (the engine adds `nodeId` / `model` / `attemptNumber`). */ | ||
| export interface CostUpdate { | ||
| readonly inputTokens: number; | ||
| readonly outputTokens: number; | ||
| readonly costMicrocents: number; | ||
| readonly cumulativeCostMicrocents: number; | ||
| } | ||
|
|
||
| /** | ||
| * Accumulates per-attempt cost across a node/session. Call `record` once per LLM attempt (each | ||
| * fallback attempt has its own `Usage`), so the running total stays accurate across a failover. | ||
| */ | ||
| export class CostTracker { | ||
| #cumulativeMicrocents = 0; | ||
|
|
||
| /** Price one attempt's usage and fold it into the running total. */ | ||
| record(modelId: string, usage: Usage): CostUpdate { | ||
| const costMicrocents = cost(modelId, usage); | ||
| this.#cumulativeMicrocents += costMicrocents; | ||
| return { | ||
| inputTokens: usage.inputTokens, | ||
| outputTokens: usage.outputTokens, | ||
| costMicrocents, | ||
| cumulativeCostMicrocents: this.#cumulativeMicrocents, | ||
| }; | ||
| } | ||
|
|
||
| /** The running total in integer micro-cents. */ | ||
| get cumulativeCostMicrocents(): number { | ||
| return this.#cumulativeMicrocents; | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using direct property lookup
MODEL_PRICING[modelId]can lead to unexpected behavior or bypasses ifmodelIdmatches built-inObject.prototypeproperties (such as'toString','valueOf', or'constructor'). SincemodelIdis a dynamic string (potentially coming from user-authored YAML or external inputs), it is safer to verify ownership usingObject.prototype.hasOwnProperty.callbefore accessing the pricing record.