Skip to content

_build_judge_config does not forward SUT endpoint in self-judge mode #15

Description

@Aliciawque

Bug

Depends on #14 (JudgeConfig.endpoint field).

ifixai/cli/orchestrator.py, _build_judge_config — when running with --endpoint and --eval-mode self, the SUT correctly connects to the custom endpoint, but the self-judge config is built without it:

if eval_mode == "self":
    return JudgeConfig(
        provider=sut_provider or "",
        api_key=sut_api_key,
        model=sut_model,
        max_calls_per_run=max_calls,
        timeout=timeout,
    )

The judge falls back to api.anthropic.com (for anthropic) or localhost:8000 (for http), causing 401 auth errors or connection refused.

Suggested Fix

Add sut_endpoint parameter:

def _build_judge_config(
    eval_mode: str,
    sut_provider: str | None,
    sut_api_key: str,
    sut_model: str | None,
    sut_endpoint: str | None = None,  # <-- add
    ...
) -> JudgeConfig | None:
    ...
    if eval_mode == "self":
        return JudgeConfig(
            provider=sut_provider or "",
            api_key=sut_api_key,
            model=sut_model,
            endpoint=sut_endpoint,  # <-- forward
            ...
        )

And in cli/run.py:

judge_config = _build_judge_config(
    ...
    sut_endpoint=endpoint,  # <-- add
    ...
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions