Skip to content

Commit f75ea0c

Browse files
Merge branch 'main' into fix/ifix-fixture-name-directory-20261007
2 parents 0ce448b + 48cd7b3 commit f75ea0c

26 files changed

Lines changed: 1532 additions & 34 deletions

‎SECURITY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ iFixAi sends pseudonymous run telemetry — from the CLI, the plugin, and scaffo
8282
- a file at `${XDG_CONFIG_HOME:-~/.config}/ifixai/telemetry-opt-out`
8383
- automatically in CI (`CI`, or a known vendor variable such as `GITHUB_ACTIONS`)
8484

85-
**Data protection (GDPR / UK GDPR).** The install id is a persistent identifier, so this telemetry is *pseudonymous personal data*, not anonymous. Lawful basis: legitimate interest (Art 6(1)(f)) in understanding adoption and retention; the processing is minimal (seven fields, no model I/O, only classifications and never the paths they were derived from), IP-stripped, opt-out-respecting, and never used for profiling or advertising. **Retention: events are kept indefinitely** — we do not auto-delete. **Erasure:** run `ifixai run --show-id` to read your install id, then email **info@ime.life** to have its events deleted; deleting the local `install-id` file stops future linkage but does not remove rows already collected. Data controller: iMe (**info@ime.life**).
85+
**Data protection (GDPR / UK GDPR).** The install id is a persistent identifier, so this telemetry is *pseudonymous personal data*, not anonymous. Lawful basis: legitimate interest (Art 6(1)(f)) in understanding adoption and retention; the processing is minimal (seven fields, no model I/O, only classifications and never the paths they were derived from), IP-stripped, opt-out-respecting, and never used for profiling or advertising. **Retention: events are kept indefinitely** — we do not auto-delete. **Erasure:** run `ifixai run --show-id` to read your install id, then email **info@ime.life** to have its events deleted; deleting the local `install-id` file stops future linkage but does not remove rows already collected. Data controller: iFixAi (**info@ime.life**).
8686

8787
## Coordinated disclosure
8888

‎docs/cli.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,11 +73,32 @@ ifixai run -p openai -k "$OPENAI_API_KEY" -c DECEPTION # example: one category
7373
| `--concurrency`, `-j` | `5` | Max in-flight LLM requests (1-20). Overrides `IFIXAI_CONCURRENCY`. |
7474
| `--no-parallel` | off | Alias for `--concurrency 1`. |
7575
| `--dry-run` | off | Print inspection and judge-call estimates, then exit. |
76+
| `--no-promo` | off | Skip the [Pro-version offer](#pro-version-offer) and start the run immediately. `IFIXAI_NO_PROMO=1` skips it on every run. |
7677
| `--reliability-out` | `runs` | Directory for `manifest.json`, one subdir per run. |
7778
| `--run-nonce` | fresh | Replay-protection nonce (16 hex chars), recorded in the manifest. |
7879
| `IFIXAI_JUDGE_FALLBACKS` (env) | packaged JSON | Path to the judge fallback-model chain. See [Judge fallback models](#judge-fallback-models). |
7980
| `--holdout-seed`, `--b{12,14,28,29,30,32}-seed` | fresh random | Pin, or set matching `IFIXAI_*_SEED`, to replay a run. See [reproducibility.md](reproducibility.md). |
8081

82+
## Pro-version offer
83+
84+
The first interactive `ifixai run` on a machine opens with one question: whether you'd like to claim your free fast audit at <https://www.ifixai.ai/>. It's shown once, whatever you answer (Ctrl-C included); later runs start straight away. The marker is a `pro-offer-shown` file in `$XDG_CONFIG_HOME/ifixai` (else `~/.config/ifixai`): delete it to see the offer again.
85+
86+
Move with the arrow keys and press Enter:
87+
88+
- **No** (preselected, just press Enter): the run starts.
89+
- **Yes**: the site opens in your browser, then you pick **Continue the current run** (preselected) or **Abandon the run**. Abandoning exits `0` before anything happens: no provider calls, no telemetry, no run directory.
90+
91+
Ctrl-C on either question aborts without starting the run. Where the arrow-key menu cannot run (`NO_COLOR` is set), the same options are listed with numbers to type instead.
92+
93+
To start immediately, pass `--no-promo`, or set `IFIXAI_NO_PROMO=1` to skip it on every run:
94+
95+
```bash
96+
ifixai run --provider openai --no-promo
97+
export IFIXAI_NO_PROMO=1
98+
```
99+
100+
The offer is never shown when stdin or stdout is not a terminal (pipes, scripts, agent-driven runs), in CI, or on `--dry-run`, so automated runs are never blocked by it. `--quiet` does not skip it; use `--no-promo`. A runner that allocates a pseudo-terminal without setting `CI` looks like a person at a keyboard: pass `--no-promo` (or set `IFIXAI_NO_PROMO=1`) there.
101+
81102
## How a run is judged
82103

83104
The SUT (system under test) is the agent being graded via the SUT flags above; its key is never read from the environment. The judge grades the SUT's answers; a citable grade needs a judge from a second, different provider.

‎ifixai/cli/_branding.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ def print_startup_banner(version: str, *, quiet: bool = False) -> None:
109109
for line in _LOGO_LINES:
110110
click.echo(" " + _truecolor(line, _ACCENT_RGB, bold=True))
111111
click.echo()
112-
click.echo(_truecolor(f" ™ · v{version} · powered by iMe", _DIM_RGB))
112+
click.echo(_truecolor(f" ™ · v{version}", _DIM_RGB))
113113
click.echo()
114114

115115

‎ifixai/cli/compare.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
TestStatus,
1313
)
1414
from ifixai.reporting.comparison import compare_scorecards
15+
from ifixai.reporting.scorecard import render_partial_banner, render_resumed_banner
1516

1617

1718
@click.command()
@@ -28,6 +29,11 @@ def compare(baseline: str, enhanced: str) -> None:
2829
click.echo(click.style("ifixai Scorecard Comparison", bold=True))
2930
click.echo()
3031

32+
for label, result in [("Baseline", baseline_result), ("Enhanced", enhanced_result)]:
33+
for banner in [render_partial_banner(result), render_resumed_banner(result)]:
34+
if banner:
35+
click.echo(f" {label}: {banner}")
36+
3137
baseline_score = (
3238
f"{report.baseline_overall:.1%}"
3339
if report.baseline_overall is not None
@@ -113,7 +119,23 @@ def load_result_from_json(path: str) -> TestRunResult:
113119
for cs in raw.get("category_scores", [])
114120
]
115121

122+
provenance_metadata = {
123+
key: metadata[key]
124+
for key in (
125+
"evaluation_date", "specification_version", "self_judged",
126+
"judge_relation", "judge_stats",
127+
)
128+
if key in metadata
129+
}
116130
return TestRunResult(
131+
**provenance_metadata,
132+
partial=raw.get("partial", False),
133+
abort_reason=raw.get("abort_reason"),
134+
not_run_test_ids=raw.get("not_run_test_ids", []),
135+
resumed_run_id=raw.get("resumed_run_id"),
136+
reused_result_count=raw.get("reused_result_count", 0),
137+
warnings=raw.get("warnings", []),
138+
validation_warnings=raw.get("validation_warnings", []),
117139
system_name=metadata.get("system_name", ""),
118140
system_version=metadata.get("system_version", "1.0"),
119141
provider=metadata.get("provider", ""),

‎ifixai/cli/config_file.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
import yaml
88
from pydantic import BaseModel, ValidationError
99

10+
from ifixai.cli.reports import _write_report_atomic
11+
1012
CONFIG_FILENAME = "ifixai.yaml"
1113

1214

@@ -54,6 +56,8 @@ def load_config(start_dir: Path | None = None) -> RunConfig | None:
5456
return None
5557
try:
5658
raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
59+
except OSError as exc:
60+
raise ValueError(f"Could not read {CONFIG_FILENAME}: {exc}") from exc
5761
except yaml.YAMLError as exc:
5862
raise ValueError(f"{CONFIG_FILENAME} is not valid YAML: {exc}") from exc
5963
if not isinstance(raw, dict):
@@ -66,7 +70,8 @@ def load_config(start_dir: Path | None = None) -> RunConfig | None:
6670

6771

6872
def write_config(config: RunConfig, start_dir: Path | None = None) -> Path:
69-
"""Write ``config`` to ``ifixai.yaml`` and return the path."""
73+
"""Atomically save ``config`` to ``ifixai.yaml`` and return its requested path."""
7074
path = config_path(start_dir)
71-
path.write_text(config.to_yaml(), encoding="utf-8")
75+
# Keep linked configurations linked while replacing their resolved target.
76+
_write_report_atomic(path.resolve(), config.to_yaml())
7277
return path

‎ifixai/cli/pro_offer.py‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
"""Pro-version offer shown at the start of an interactive ``ifixai run``.
2+
3+
The offer is made once per machine, before telemetry and before any provider
4+
call, so a run the user abandons for the website leaves no trace beyond the
5+
``pro-offer-shown`` marker. It is never shown when stdin or stdout is not a
6+
terminal, or in CI, so scripted and agent-driven runs cannot block on it. Skip
7+
it with ``--no-promo`` or ``IFIXAI_NO_PROMO=1``.
8+
"""
9+
10+
import sys
11+
import webbrowser
12+
from enum import Enum
13+
from typing import TextIO, TypedDict
14+
15+
import click
16+
17+
from ifixai import telemetry
18+
from ifixai.cli import ui
19+
20+
PRO_URL = "https://www.ifixai.ai/"
21+
NO_PROMO_ENV_VAR = "IFIXAI_NO_PROMO"
22+
OFFER_MARKER_NAME = "pro-offer-shown"
23+
24+
YES_OPTION = "Yes"
25+
NO_OPTION = "No"
26+
CONTINUE_OPTION = "Continue the current run"
27+
ABANDON_OPTION = "Abandon the run"
28+
29+
30+
class RunDecision(str, Enum):
31+
"""What the user wants done with the current run after seeing the offer."""
32+
33+
CONTINUE = "continue"
34+
ABANDON = "abandon"
35+
36+
37+
class ChoiceQuestion(TypedDict):
38+
"""One question with a fixed set of answers; Enter alone picks ``default``."""
39+
40+
message: str
41+
options: tuple[str, ...]
42+
default: str
43+
44+
45+
PRO_INTEREST_QUESTION = ChoiceQuestion(
46+
message="Claim your free fast audit now?",
47+
options=(YES_OPTION, NO_OPTION),
48+
default=NO_OPTION,
49+
)
50+
RUN_DECISION_QUESTION = ChoiceQuestion(
51+
message="What would you like to do with this run?",
52+
options=(CONTINUE_OPTION, ABANDON_OPTION),
53+
default=CONTINUE_OPTION,
54+
)
55+
56+
57+
def offer_pro_version() -> RunDecision:
58+
"""Offer the Pro version and report whether the current run should go ahead.
59+
60+
Opens the Pro website when the user accepts. Returns ``CONTINUE`` without
61+
asking anything when nobody is at a terminal to answer or it was already shown.
62+
"""
63+
if not can_prompt_user() or not mark_offer_shown():
64+
return RunDecision.CONTINUE
65+
print_offer()
66+
if ask_choice(PRO_INTEREST_QUESTION) != YES_OPTION:
67+
return RunDecision.CONTINUE
68+
open_pro_site()
69+
if ask_choice(RUN_DECISION_QUESTION) == ABANDON_OPTION:
70+
return RunDecision.ABANDON
71+
return RunDecision.CONTINUE
72+
73+
74+
def can_prompt_user() -> bool:
75+
"""True only when a person is at a terminal to answer the offer."""
76+
return (
77+
is_terminal(sys.stdin) and is_terminal(sys.stdout) and not telemetry.in_ci()
78+
)
79+
80+
81+
def mark_offer_shown() -> bool:
82+
"""Create the shown-once marker; False if it exists or can't be written.
83+
84+
Marked before asking, so any answer (Ctrl-C included) counts. When it can't
85+
be written the offer is skipped, so a looping script never stops on it twice.
86+
"""
87+
try:
88+
(telemetry._ensure_config_dir() / OFFER_MARKER_NAME).touch(exist_ok=False)
89+
except OSError:
90+
return False
91+
return True
92+
93+
94+
def is_terminal(stream: TextIO | None) -> bool:
95+
"""True when the stream exists and is attached to a terminal.
96+
97+
Python sets a standard stream to ``None`` when the process has no handle
98+
for it (a closed descriptor, ``pythonw``, a detached Windows process).
99+
"""
100+
return stream is not None and stream.isatty()
101+
102+
103+
def print_offer() -> None:
104+
"""Print the Pro-version pitch and its link."""
105+
click.echo(click.style(" iFixAi Pro", bold=True))
106+
click.echo(" Claim your free fast audit")
107+
click.echo(click.style(f" {PRO_URL}", fg="cyan"))
108+
click.echo()
109+
110+
111+
def open_pro_site() -> None:
112+
"""Open the Pro website in the default browser; print the link when that fails."""
113+
if webbrowser.open(PRO_URL):
114+
click.echo(click.style(f"Opened {PRO_URL} in your browser.", fg="green"))
115+
return
116+
click.echo(
117+
click.style(
118+
f"Could not open a browser. Visit {PRO_URL} to claim your free fast audit.",
119+
fg="yellow",
120+
)
121+
)
122+
123+
124+
def ask_choice(question: ChoiceQuestion) -> str:
125+
"""Ask with an arrow-key menu, or a numbered list where menus are unavailable."""
126+
if ui.is_interactive():
127+
return ui.select_or_abort(
128+
question["message"], question["options"], question["default"]
129+
)
130+
return ask_numbered_choice(question)
131+
132+
133+
def ask_numbered_choice(question: ChoiceQuestion) -> str:
134+
"""List the options with numbers and read the number the user types.
135+
136+
The plain path for terminals where the arrow-key menu cannot run: ``NO_COLOR``
137+
is set, or ``questionary`` is not installed.
138+
"""
139+
options = question["options"]
140+
for number, option in enumerate(options, start=1):
141+
click.echo(f" [{number}] {option}")
142+
typed_number = click.prompt(
143+
question["message"],
144+
type=click.IntRange(1, len(options)),
145+
default=options.index(question["default"]) + 1,
146+
)
147+
return options[typed_number - 1]

‎ifixai/cli/run.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
_resolve_standard_eval_mode,
3636
execute_tests,
3737
)
38+
from ifixai.cli.pro_offer import NO_PROMO_ENV_VAR, RunDecision, offer_pro_version
3839
from ifixai.cli.reports import save_reports
3940
from ifixai.cli.schemas import InteractiveConfig
4041
from ifixai.core.concurrency import (
@@ -143,6 +144,7 @@ def _governance_source_warning(source: str) -> str | None:
143144
"huggingface",
144145
"http",
145146
"langchain",
147+
"litellm",
146148
"openrouter",
147149
"orcarouter",
148150
"requesty",
@@ -733,6 +735,16 @@ def _validate_min_score(ctx: click.Context, param: click.Parameter, value: float
733735
help="Disable pseudonymous run telemetry for this run. Set IFIXAI_TELEMETRY=0 or "
734736
"DO_NOT_TRACK=1 to disable it permanently.",
735737
)
738+
@click.option(
739+
"--no-promo",
740+
"no_promo",
741+
is_flag=True,
742+
default=False,
743+
envvar=NO_PROMO_ENV_VAR,
744+
help="Skip the Pro-version offer and start the run immediately. Set "
745+
f"{NO_PROMO_ENV_VAR}=1 to skip it on every run. The offer is never shown "
746+
"in CI or when input/output is not a terminal.",
747+
)
736748
@click.option(
737749
"--print-telemetry",
738750
"print_telemetry",
@@ -795,6 +807,7 @@ def run(
795807
grounding: str,
796808
quiet: bool,
797809
no_telemetry: bool,
810+
no_promo: bool,
798811
print_telemetry: bool,
799812
show_install_id: bool,
800813
) -> None:
@@ -937,6 +950,10 @@ def run(
937950
holdout_seed = resume_manifest.holdout_seed
938951

939952
print_startup_banner(IFIXAI_VERSION, quiet=quiet)
953+
# Asked before telemetry and any provider call, so abandoning leaves no trace.
954+
if not (no_promo or dry_run) and offer_pro_version() is RunDecision.ABANDON:
955+
click.echo("Run abandoned -- no inspections were run.")
956+
return
940957
if no_telemetry:
941958
telemetry.disable()
942959
telemetry.show_disclosure()

‎ifixai/cli/scaffold.py‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -144,17 +144,21 @@ def _splice_agents_md(existing: str, body: str) -> str:
144144
if has_begin and has_end:
145145
pre = existing.split(MD_BEGIN, 1)[0]
146146
post = existing.split(MD_END, 1)[1]
147-
return f"{pre.rstrip()}\n\n{block}\n{post.lstrip()}".strip() + "\n"
148-
sep = "" if not existing.strip() else existing.rstrip() + "\n\n"
149-
return f"{sep}{block}\n"
147+
return pre + block + post
148+
separator = "" if not existing or existing.endswith("\n\n") else "\n" if existing.endswith("\n") else "\n\n"
149+
return existing + separator + block + "\n"
150150

151151

152152
def _strip_agents_md(existing: str) -> str:
153153
if MD_BEGIN not in existing or MD_END not in existing:
154154
return existing
155155
pre = existing.split(MD_BEGIN, 1)[0]
156156
post = existing.split(MD_END, 1)[1]
157-
return (pre.rstrip() + "\n" + post.lstrip()).strip() + "\n" if (pre.strip() or post.strip()) else ""
157+
if post.startswith("\r\n"):
158+
post = post[2:]
159+
elif post.startswith("\n"):
160+
post = post[1:]
161+
return pre + post
158162

159163

160164
def _resolve_agents(agents: str | None, project_root: Path, name: str) -> list[AgentTarget]:
@@ -233,8 +237,13 @@ def install(agents: str | None, name: str, project_dir: str, revert: bool, list_
233237
path.parent.mkdir(parents=True, exist_ok=True)
234238

235239
if t.fmt == "agents_md":
236-
existing = path.read_text(encoding="utf-8") if path.is_file() else ""
237-
path.write_text(_splice_agents_md(existing, body), encoding="utf-8")
240+
existing = ""
241+
if path.is_file():
242+
with path.open(encoding="utf-8", newline="") as handle:
243+
existing = handle.read()
244+
updated = _splice_agents_md(existing, body)
245+
with path.open("w", encoding="utf-8", newline="") as handle:
246+
handle.write(updated)
238247
click.echo(f" {click.style('updated', fg='green')} {t.label}: {path}")
239248
written += 1
240249
continue
@@ -266,9 +275,12 @@ def _do_revert(targets: list[AgentTarget], project_root: Path, name: str) -> Non
266275
text = path.read_text(encoding="utf-8")
267276

268277
if t.fmt == "agents_md":
278+
with path.open(encoding="utf-8", newline="") as handle:
279+
text = handle.read()
269280
stripped = _strip_agents_md(text)
270-
if stripped.strip():
271-
path.write_text(stripped, encoding="utf-8")
281+
if stripped:
282+
with path.open("w", encoding="utf-8", newline="") as handle:
283+
handle.write(stripped)
272284
else:
273285
path.unlink()
274286
click.echo(f" {click.style('reverted', fg='yellow')} {t.label}: {path}")

0 commit comments

Comments
 (0)