Skip to content

Commit 7e4c165

Browse files
n-papaioannouclaude
andcommitted
docs: clarify README and fix accuracy nits
- Add Wiring governance to TOC; reorder providers extras table to match install order in Quick start. - Pull Judge selection rules to the top of Quick start so they apply to every provider example below. - Restructure provider sections so each example has a coherent judge setup (cross-judge env in Anthropic/Gemini/Bedrock/HF/HTTP, explicit --judge-* flags for OpenRouter/Azure, single-key self-judge for LangChain). - Fix typo: "Fives" -> "Five" example fixtures. - Drop bogus "90 lines" count for smoke_tiny.yaml (it's 88). - Tighten "ifixai list fixtures" comment in CLI reference -- the command only lists registered named fixtures; example fixtures load by path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 13afffe commit 7e4c165

1 file changed

Lines changed: 40 additions & 29 deletions

File tree

‎README.md‎

Lines changed: 40 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -56,13 +56,14 @@ and track over time.
5656
5. [Five scorecard pillars](#five-scorecard-pillars)
5757
6. [Domain-neutral fixtures](#domain-neutral-fixtures)
5858
7. [Author your own fixture](#author-your-own-fixture)
59-
8. [Supported providers](#supported-providers)
60-
9. [CLI reference](#cli-reference)
61-
10. [Scoring](#scoring)
62-
11. [Python API](#python-api)
63-
12. [Development](#development)
64-
13. [Contact](#contact)
65-
14. [License](#license)
59+
8. [Wiring governance](#wiring-governance)
60+
9. [Supported providers](#supported-providers)
61+
10. [CLI reference](#cli-reference)
62+
11. [Scoring](#scoring)
63+
12. [Python API](#python-api)
64+
13. [Development](#development)
65+
14. [Contact](#contact)
66+
15. [License](#license)
6667

6768
## Requirements
6869

@@ -73,10 +74,10 @@ and track over time.
7374
|---|---|---|
7475
| *(none)* | Core only | `mock`, `http`, `langchain` (you must `pip install langchain` yourself) |
7576
| `openai` | `openai` SDK | `openai` |
76-
| `azure` | `openai` SDK | `azure` (same client; set `--endpoint` to your Azure OpenAI resource) |
77-
| `openrouter` | `openai` SDK (OpenRouter exposes an OpenAI-compatible endpoint; any compatible SDK or `--provider http` also works) | `openrouter` |
7877
| `anthropic` | `anthropic` SDK | `anthropic` |
78+
| `openrouter` | `openai` SDK (OpenRouter exposes an OpenAI-compatible endpoint; any compatible SDK or `--provider http` also works) | `openrouter` |
7979
| `gemini` | `google-generativeai` | `gemini` |
80+
| `azure` | `openai` SDK | `azure` (same client; set `--endpoint` to your Azure OpenAI resource) |
8081
| `bedrock` | `boto3` | `bedrock` |
8182
| `huggingface` | `huggingface-hub` | `huggingface` |
8283
| `dev` | Lint, types, tests, security | [Contributing](CONTRIBUTING.md) only |
@@ -96,6 +97,12 @@ The CLI does **not** auto-read the SUT API key from the environment: pass **`--a
9697

9798
Omitting `--fixture` uses the built-in **default** fixture. Runs emit a scorecard under `./ifixai-results/` (override with `--output`). Typical wall time is a few minutes on broadband.
9899

100+
**Judge selection:**
101+
- **Default:** judge = any non-SUT provider key in your env, run on that provider's default model.
102+
- **Multiple keys:** tiebreaker order is `anthropic → openai → gemini → openrouter → azure → bedrock → huggingface`.
103+
- **No non-SUT key:** pass `--eval-mode self`, or the run refuses.
104+
- **Override:** `--judge-provider` / `--judge-api-key` / `--judge-model`.
105+
99106
### 0 — Mock (no cloud keys)
100107

101108
```bash
@@ -120,34 +127,37 @@ Single key only (self-judge):
120127
ifixai run --provider openai --api-key "$OPENAI_API_KEY" --eval-mode self
121128
```
122129

123-
### 2 — OpenRouter
130+
### 2 — Anthropic
124131

125132
```bash
126-
pip install -e ".[openrouter]" # installs openai SDK; OpenRouter is OpenAI-compatible — other compatible SDKs or --provider http work too
127-
export OPENROUTER_API_KEY=sk-or-...
133+
pip install -e ".[anthropic]"
128134
export ANTHROPIC_API_KEY=sk-ant-api03-...
129-
ifixai run --provider openrouter --api-key "$OPENROUTER_API_KEY" --model openai/gpt-4o
135+
export GEMINI_API_KEY=... # second provider for cross-judge (or use --eval-mode self)
136+
ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --model claude-sonnet-4-20250514
130137
```
131138

132-
### 3 — Anthropic
139+
### 3 — OpenRouter (explicit judge)
133140

134141
```bash
135-
pip install -e ".[anthropic]"
142+
pip install -e ".[openrouter]" # installs openai SDK; OpenRouter is OpenAI-compatible — other compatible SDKs or --provider http work too
143+
export OPENROUTER_API_KEY=sk-or-...
136144
export ANTHROPIC_API_KEY=sk-ant-api03-...
137-
export OPENAI_API_KEY=sk-...
138-
ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --model claude-sonnet-4-20250514
145+
ifixai run --provider openrouter --api-key "$OPENROUTER_API_KEY" --model openai/gpt-4o \
146+
--judge-provider anthropic --judge-api-key "$ANTHROPIC_API_KEY" --judge-model claude-sonnet-4-20250514
139147
```
140148

149+
Pinning the judge avoids the underlying-model collision OpenRouter routing can introduce (e.g. routing the SUT to an Anthropic model while Anthropic is also the auto-judge).
150+
141151
### 4 — Google Gemini
142152

143153
```bash
144154
pip install -e ".[gemini]"
145155
export GEMINI_API_KEY=... # or GOOGLE_API_KEY
146-
export OPENAI_API_KEY=sk-...
156+
export ANTHROPIC_API_KEY=sk-ant-api03-... # second provider for cross-judge (or use --eval-mode self)
147157
ifixai run --provider gemini --api-key "$GEMINI_API_KEY"
148158
```
149159

150-
### 5 — Azure OpenAI
160+
### 5 — Azure OpenAI (explicit judge)
151161

152162
```bash
153163
pip install -e ".[azure]" # or .[openai] — same OpenAI-compatible SDK
@@ -156,7 +166,8 @@ export ANTHROPIC_API_KEY=sk-ant-api03-...
156166
ifixai run --provider azure \
157167
--endpoint https://YOUR_RESOURCE.openai.azure.com/ \
158168
--api-key "$AZURE_OPENAI_API_KEY" \
159-
--model YOUR_DEPLOYMENT_NAME
169+
--model YOUR_DEPLOYMENT_NAME \
170+
--judge-provider anthropic --judge-api-key "$ANTHROPIC_API_KEY" --judge-model claude-sonnet-4-20250514
160171
```
161172

162173
### 6 — AWS Bedrock
@@ -165,7 +176,7 @@ ifixai run --provider azure \
165176
pip install -e ".[bedrock]"
166177
export AWS_ACCESS_KEY_ID=...
167178
export AWS_SECRET_ACCESS_KEY=...
168-
export OPENAI_API_KEY=sk-...
179+
export GEMINI_API_KEY=... # second provider for cross-judge (or use --eval-mode self)
169180
ifixai run --provider bedrock --api-key not-used \
170181
--model anthropic.claude-3-5-sonnet-20240620-v1:0
171182
```
@@ -177,7 +188,7 @@ Authentication uses the **standard AWS credential chain** (env vars or instance
177188
```bash
178189
pip install -e ".[huggingface]"
179190
export HF_TOKEN=hf_...
180-
export OPENAI_API_KEY=sk-...
191+
export ANTHROPIC_API_KEY=sk-ant-api03-... # second provider for cross-judge (or use --eval-mode self)
181192
ifixai run --provider huggingface --api-key "$HF_TOKEN" --model meta-llama/Llama-3.1-8B-Instruct
182193
```
183194

@@ -187,7 +198,7 @@ ifixai run --provider huggingface --api-key "$HF_TOKEN" --model meta-llama/Llama
187198

188199
```bash
189200
pip install -e "."
190-
export OPENAI_API_KEY=sk-...
201+
export GEMINI_API_KEY=... # second provider for cross-judge (or use --eval-mode self)
191202
ifixai run --provider http \
192203
--endpoint http://localhost:8000/v1 \
193204
--api-key YOUR_SERVER_TOKEN \
@@ -196,13 +207,13 @@ ifixai run --provider http \
196207

197208
Optional JSON headers: set **`IFIXAI_EXTRA_HEADERS`** to a JSON object (see `ifixai/providers/http.py`).
198209

199-
### 9 — LangChain
210+
### 9 — LangChain (single-key self-judge)
200211

201212
```bash
202213
pip install -e "."
203214
pip install langchain # not bundled as a named extra
204-
export OPENAI_API_KEY=sk-...
205-
ifixai run --provider langchain --api-key "$OPENAI_API_KEY"
215+
export OPENAI_API_KEY=sk-... # one key only — SUT and judge share the same model
216+
ifixai run --provider langchain --api-key "$OPENAI_API_KEY" --eval-mode self
206217
```
207218

208219
Wire your chain inside the LangChain adapter as documented in the provider module.
@@ -269,7 +280,7 @@ attestation facility (no inspections use it today), B28 RAG context integrity, a
269280
## Domain-neutral fixtures
270281

271282
Test code is domain-neutral. Industry knowledge lives in user-authored
272-
fixture YAML — never in test code. Fives example fixtures live under
283+
fixture YAML — never in test code. Five example fixtures live under
273284
[`ifixai/fixtures/examples/`](ifixai/fixtures/examples/):
274285

275286
```bash
@@ -292,7 +303,7 @@ Your domain knowledge (roles, users, tools, permissions, policies) lives in
292303
a fixture file (YAML or JSON). The fastest path:
293304

294305
```bash
295-
# Start from the smallest valid fixture (90 lines, every required key populated)
306+
# Start from the smallest valid fixture (every required key populated)
296307
cp ifixai/fixtures/smoke_tiny.yaml my-fixture.yaml
297308

298309
# Edit roles, users, tools, permissions to match your system
@@ -375,7 +386,7 @@ ifixai init # check env for provider keys, suggest a first ru
375386
ifixai run # run tests (Standard or Full mode)
376387
ifixai run --fixture FILE # run with a custom fixture (YAML or JSON)
377388
ifixai list tests # list all 32 tests
378-
ifixai list fixtures # list built-in fixtures
389+
ifixai list fixtures # list registered named fixtures (examples/ are loaded by path)
379390
ifixai validate # validate the per-test layout (32 folders)
380391
ifixai validate FILE # validate a fixture against schema.json
381392
ifixai compare A B # diff two scorecard reports

0 commit comments

Comments
 (0)