Skip to content

fix(cli): accept the registered LiteLLM provider - #200

Merged
n-papaioannou merged 2 commits into
ifixai-ai:mainfrom
rudycelekli:fix/ifix-litellm-cli-choice-20261006
Oct 7, 2026
Merged

n-papaioannou merged 2 commits into
ifixai-ai:mainfrom
rudycelekli:fix/ifix-litellm-cli-choice-20261006

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

The resolver and optional dependency register litellm, but the CLI provider choice omits it. Both --provider litellm and --judge-provider litellm are rejected before the registered provider can run, and saved configurations fail the same choice validation. Add LiteLLM to the shared provider choice used by SUT and judge flags.

Test plan:

  • Production CLI SUT, judge and saved-config paths: 3 failing regressions on current main, all 7 tests in the existing judge configuration module passing after the change. Uppercase flags also use the existing case-insensitive choice contract.
  • Dry-run reproductions use synthetic credentials and make no provider requests.
  • Ruff 0.16.9, Bandit, layout, all eleven shipped example fixtures and diff check pass. Mypy is advisory and exits 1: Found 213 errors in 69 files (checked 378 source files).

Worked CLI example with native LiteLLM 1.82.1/OpenAI 2.41.0 SDKs against an owned loopback server and the built-in mock judge (illustrative transport plumbing; no paid request or model-quality claim):

{
  "provider": "litellm",
  "sut_transport": "native LiteLLM SDK -> controlled loopback HTTP",
  "judge_provider": "mock",
  "local_http_requests": 12,
  "test_results": [
    {
      "test_id": "B07",
      "score": 1.0,
      "status": "pass",
      "passing": true,
      "evidence_count": 10
    }
  ],
  "overall_score": null,
  "cli_exit": 2
}

The server injected one synthetic HTTP 500, then returned controlled responses. The overall score remains null and the CLI exits 2 because B01/B08/P01 were outside this selected B07 run.

Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
@n-papaioannou
n-papaioannou merged commit 29d7ad2 into ifixai-ai:main Oct 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants