Skip to content

Commit 15d997e

Browse files
n-papaioannouclaude
andcommitted
chore: initial public release
Initial public release of ifixai diagnostic, sourced from the ime_benchmarks working tree. Excludes local-only artefacts (tests/, RELEASING.md, .gitignore, specs/, caches, env files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0 parents  commit 15d997e

214 files changed

Lines changed: 17771 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
branches: [main, "fix/**"]
7+
8+
jobs:
9+
test:
10+
runs-on: ubuntu-latest
11+
strategy:
12+
matrix:
13+
python-version: ["3.10", "3.11", "3.12"]
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Python ${{ matrix.python-version }}
19+
uses: actions/setup-python@v5
20+
with:
21+
python-version: ${{ matrix.python-version }}
22+
23+
- name: Install dependencies
24+
run: python -m pip install -e ".[dev]"
25+
26+
- name: Lint
27+
run: ruff check ifixai tests
28+
29+
- name: Test with coverage
30+
run: pytest tests/ ifixai/tests/ --cov=ifixai --cov-report=term-missing --cov-fail-under=60 -q --tb=short -m "not integration"
31+
32+
- name: Integration tests (disk/fixture-dependent)
33+
run: pytest tests/ -q --tb=short -m integration
34+
35+
- name: Security scan
36+
run: bandit -r ifixai -ll
37+
38+
- name: Docs presence check
39+
run: |
40+
test -f CHANGELOG.md || { echo "CHANGELOG.md missing"; exit 1; }
41+
test -f CONTRIBUTING.md || { echo "CONTRIBUTING.md missing"; exit 1; }
42+
test -f SECURITY.md || { echo "SECURITY.md missing"; exit 1; }
43+
echo "governance docs present"
44+
45+
inspect-smoke:
46+
runs-on: ubuntu-latest
47+
needs: test
48+
if: ${{ github.event_name == 'pull_request' }}
49+
steps:
50+
- uses: actions/checkout@v4
51+
- uses: actions/setup-python@v5
52+
with:
53+
python-version: "3.12"
54+
- name: Check OpenAI key presence
55+
id: check_key
56+
env:
57+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
58+
run: |
59+
if [ -n "$OPENAI_API_KEY" ]; then
60+
echo "present=true" >> "$GITHUB_OUTPUT"
61+
else
62+
echo "present=false" >> "$GITHUB_OUTPUT"
63+
fi
64+
- name: Install with inspect extras
65+
if: steps.check_key.outputs.present == 'true'
66+
run: python -m pip install -e ".[dev,inspect]"
67+
- name: Inspect B01 smoke
68+
if: steps.check_key.outputs.present == 'true'
69+
env:
70+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
71+
run: inspect eval ifixai/inspect_integration/tasks.py@ifixai_b01

‎.pre-commit-config.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
repos:
2+
- repo: https://github.com/pre-commit/pre-commit-hooks
3+
rev: v5.0.0
4+
hooks:
5+
- id: check-added-large-files
6+
- id: detect-private-key
7+
- id: end-of-file-fixer
8+
- id: trailing-whitespace
9+
- id: check-yaml
10+
- id: check-toml
11+
12+
- repo: https://github.com/astral-sh/ruff-pre-commit
13+
rev: v0.6.9
14+
hooks:
15+
- id: ruff
16+
args: [--fix, --exit-non-zero-on-fix]
17+
18+
- repo: https://github.com/gitleaks/gitleaks
19+
rev: v8.21.2
20+
hooks:
21+
- id: gitleaks
22+
23+
- repo: local
24+
hooks:
25+
- id: no-env-files
26+
name: block .env files from being committed
27+
entry: refusing to commit .env files; use .env.example for tracked templates
28+
language: fail
29+
files: '(^|/)\.env($|\.)'
30+
exclude: '\.env\.(example|sample|template)$'

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Changelog
2+
3+
All notable changes to `ifixai` will be recorded here. Format follows
4+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project uses
5+
[SemVer](https://semver.org/spec/v2.0.0.html).
6+
7+
## [1.0.0] — 2026-04-27
8+
9+
Initial public release.

‎CLAUDE.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
CLAUDE.md
2+
Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.
3+
4+
Tradeoff: These guidelines bias toward caution over speed. For trivial tasks, use judgment.
5+
6+
0. Answer Style
7+
Concise and direct. No long paragraphs.
8+
9+
Prefer bullets for any answer with more than one point.
10+
Pair explanations with a concrete example (command, snippet, path).
11+
No throat-clearing ("Great question", "Let me explain", "In summary"). Start with the answer.
12+
13+
1. Think Before Coding
14+
Don't assume. Don't hide confusion. Surface tradeoffs.
15+
16+
Before implementing:
17+
18+
State your assumptions explicitly. If uncertain, ask.
19+
If multiple interpretations exist, present them - don't pick silently.
20+
If a simpler approach exists, say so. Push back when warranted.
21+
If something is unclear, stop. Name what's confusing. Ask.
22+
2. Simplicity First
23+
Minimum code that solves the problem. Nothing speculative.
24+
25+
No features beyond what was asked.
26+
No abstractions for single-use code.
27+
No "flexibility" or "configurability" that wasn't requested.
28+
No error handling for impossible scenarios.
29+
If you write 200 lines and it could be 50, rewrite it.
30+
Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
31+
32+
3. Surgical Changes
33+
Touch only what you must. Clean up only your own mess.
34+
35+
When editing existing code:
36+
37+
Don't "improve" adjacent code, comments, or formatting.
38+
Don't refactor things that aren't broken.
39+
Match existing style, even if you'd do it differently.
40+
If you notice unrelated dead code, mention it - don't delete it.
41+
When your changes create orphans:
42+
43+
Remove imports/variables/functions that YOUR changes made unused.
44+
Don't remove pre-existing dead code unless asked.
45+
The test: Every changed line should trace directly to the user's request.
46+
47+
4. Goal-Driven Execution
48+
Define success criteria. Loop until verified.
49+
50+
Transform tasks into verifiable goals:
51+
52+
"Add validation" → "Write tests for invalid inputs, then make them pass"
53+
"Fix the bug" → "Write a test that reproduces it, then make it pass"
54+
"Refactor X" → "Ensure tests pass before and after"
55+
For multi-step tasks, state a brief plan:
56+
57+
1. [Step] → verify: [check]
58+
2. [Step] → verify: [check]
59+
3. [Step] → verify: [check]
60+
Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
61+
62+
<!-- SPECKIT START -->
63+
Active feature plan: [specs/015-manifest-judge-cleanup/plan.md](specs/015-manifest-judge-cleanup/plan.md)
64+
Spec: [specs/015-manifest-judge-cleanup/spec.md](specs/015-manifest-judge-cleanup/spec.md)
65+
<!-- SPECKIT END -->

‎CONTRIBUTING.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Contributing to ifixai
2+
3+
Thanks for considering a contribution. This guide covers the mechanics of adding inspections, fixtures, providers, and running the test suite. For the behavioural contract (what the project expects from the code you write), see the root `CLAUDE.md` if it exists plus `.claude/rules/common/*.md` in the repository.
4+
5+
## Environment setup
6+
7+
```bash
8+
git clone <your-fork-url> ifixai
9+
cd ifixai
10+
python -m venv .venv
11+
source .venv/bin/activate
12+
pip install -e ".[dev]"
13+
pre-commit install
14+
```
15+
16+
The `pre-commit install` step wires up local hooks (`gitleaks`, `ruff`, a `.env` guard, and a sanitizer regex for known-internal identifiers). Run them on demand with `pre-commit run --all-files`.
17+
18+
Verify:
19+
20+
```bash
21+
ruff check ifixai tests
22+
python -m pytest --cov=ifixai --cov-report=term
23+
```
24+
25+
The coverage floor is declared in `.github/workflows/ci.yml`. The policy is simple: the floor is pinned at `floor(current_baseline) - 2` percentage points so a flaky test cannot accidentally erode coverage, and is ratcheted upward when a sustained improvement lands. Drop-below fails CI.
26+
27+
## Adding a test (inspection)
28+
29+
Each inspection is one file under `ifixai/tests/bNN_short_name.py`. The minimum contract:
30+
31+
1. Declare the `SPEC` — a `InspectionSpec` instance with `test_id`, `name`, `category` (one of the five `InspectionCategory` values), `description`, `threshold`, `weight`, `scoring_method`, and optional `is_strategic` / `is_mandatory_minimum` flags.
32+
2. Implement a subclass of `BaseTest` (from `ifixai.tests.base`). Override `run()` to produce a list of `EvidenceItem`s. Use `self.pipeline.evaluate(...)` to get a pass/fail from the configured judge.
33+
3. Declare `required_fixture_keys: frozenset[str]` on the subclass listing every fixture key the inspection's templates reference. The fixture loader validates this at load time; inspections that reference keys the fixture doesn't provide fail fast with an actionable error.
34+
4. Render every prompt through `ifixai.utils.template_renderer.render(template, context)`. Direct `str.format(...)` or f-string interpolation on fixture values is forbidden — it silently leaks `{placeholder}` literals to the model when a key is missing.
35+
5. Register the inspection in `ifixai/tests/registry.py` (import + add to `ALL_SPECS` + `create_inspection` switch).
36+
6. Update `ifixai/scoring/category_weights.py` only if the inspection belongs to the strategic set.
37+
38+
### Inspection testing
39+
40+
Every inspection should have a companion test under `tests/test_bNN_*.py` covering at least:
41+
42+
- `test_{inspection}_spec_invariants_unchanged` — asserts immutable fields (id, category, threshold, weight, strategic-ness).
43+
- Happy-path evidence generation against a fixture.
44+
- Failure path (provider error, empty response, malformed output).
45+
46+
## Authoring a fixture
47+
48+
Fixtures are YAML files under `ifixai/fixtures/`. Validate against `ifixai/fixtures/schema.json`. A fixture MUST supply every key listed in the union of every registered inspection's `required_fixture_keys`.
49+
50+
The `x-placeholders` section of the schema (lint-only) enumerates the placeholder keys any inspection may reference. Keep it in sync when you add a inspection that references a new key.
51+
52+
Three example fixtures live under `ifixai/fixtures/examples/`. Copy one as a starting point.
53+
54+
## Registering a new provider
55+
56+
Providers implement the `ChatProvider` protocol from `ifixai/providers/base.py`. Steps:
57+
58+
1. Create `ifixai/providers/<your_provider>.py` implementing at minimum `send_message` — other capability methods may raise `NotImplementedError` if the provider does not expose them.
59+
2. Register the provider string in `ifixai/providers/resolver.py`.
60+
3. Add an optional dependency extra in `pyproject.toml` under `[project.optional-dependencies]` so users install only what they need.
61+
4. Do not swallow exceptions silently. If the provider has idiomatic error types, translate them into `ProviderError` (or a subclass).
62+
5. Add `tests/test_<your_provider>.py` with stub / mock coverage.
63+
64+
## Running tests locally
65+
66+
```bash
67+
# full suite
68+
python -m pytest
69+
70+
# one module
71+
python -m pytest tests/test_your_thing.py
72+
73+
# with coverage
74+
python -m pytest --cov=ifixai --cov-report=term-missing
75+
76+
# ruff
77+
ruff check ifixai tests
78+
79+
# type check
80+
mypy ifixai
81+
82+
# security scan
83+
bandit -r ifixai -ll
84+
```
85+
86+
## Commit conventions
87+
88+
Follow the Conventional Commits-style prefixes used by this project:
89+
90+
- `feat:` new user-visible feature
91+
- `fix:` bug fix
92+
- `refactor:` behaviour-preserving change
93+
- `docs:` documentation only
94+
- `test:` test-only changes
95+
- `chore:` tooling / housekeeping
96+
- `perf:` performance improvement
97+
- `ci:` CI configuration
98+
99+
Keep commits small and atomic. Include a test with every `fix:` or `feat:` that changes observable behaviour.
100+
101+
## Pull requests
102+
103+
- Target branch: `main`.
104+
- Include a test plan in the PR body.
105+
- Confirm `ruff`, `pytest`, and `bandit` all pass locally before requesting review. `mypy` is advisory (run it locally if you touched typed surfaces, but it is not a CI gate).
106+
- For any inspection / fixture / provider change, paste one worked example scorecard snippet (JSON or Markdown) into the PR body. The `ifixai-results/` directory is gitignored and cannot be updated as part of a PR.
107+
108+
## Where to ask
109+
110+
Open a GitHub issue on the repository for questions, bug reports, or feature proposals. For security-sensitive reports see `SECURITY.md`.

0 commit comments

Comments
 (0)