(bug) Model picker allows invalid harness/model/auth combinations without validation

[mmabrouk]

Symptom. In the agent model picker, choosing the Pi harness with the Claude Haiku model and subscription auth saves without any warning. Every run against that saved configuration then fails at runtime with "model not available" — the user has a config that looks valid in the UI and only discovers it's broken when they try to use it.

Cause, where known. There is no validation at selection/save time that cross-checks harness + model + auth-mode compatibility. The picker lets you compose combinations that the backend cannot actually run, and the failure only surfaces later, on the first live run, with a message that doesn't point back at the picker choice that caused it.

Steps to reproduce.

1. Open an agent's config in the playground.
2. In the harness selector, choose Pi.
3. In the model selector, choose Claude Haiku, with subscription ("use your own login") auth.
4. Save the configuration. No error or warning appears; the save succeeds.
5. Run the agent (send any message).

Actual: the run fails with "model not available".
Expected: either the picker disallows/flags this combination before it can be saved, or the run-time error names the actual mismatch (harness/model/auth) so the user can fix it without guessing.

Severity: High — the config silently passes validation and only fails downstream, and the failure message gives no path back to the cause.

Found during the agent-config-editing QA campaign; pre-existing on main (not specific to the agent-config-editing feature).

[linear-code]

AGE-4072

[dosubot]

This looks closely related to a cluster of issues that have been partially addressed on main, though gaps clearly remain:

Prior art on this exact problem:

• Issue #5114 reported the same symptom — selecting a model under the Pi harness that the backend couldn't run, with the picker showing no warning. That was fixed by normalizing provider-family values and adding harness/provider/deployment pair validation. The Pi harness now explicitly allows only direct and custom deployments for OpenAI-family models.
• Issue #5614 caught a related case during release QA: a DeepSeek-family model with provider: custom was correctly rejected by pi_core, confirming the validation does work for some combinations.
• Issue #5118 found that the picker displayed false "not available" badges for valid custom-provider models because availability was inferred from model-ID format rather than actual backend/harness support.

Frontend filtering that exists today: PRs #4839, #4881, and #5057 added harness-aware model filtering using capabilities from the /inspect endpoint — when you switch harness, incompatible models should be cleared or flagged. PR #5522 (currently open) extends this to vault-aware custom-provider checks.

The gap you're hitting: Backend save-time validation was proposed in PR #5104, which would reject invalid agent-template shapes (including cross-field rules like "Claude harness requires Anthropic provider") at commit and test_run time with structured HTTP 400 errors. That PR was closed without merging. Without it, invalid combos that slip past the frontend filter are only caught at runtime, and issue #5641 confirms those failures currently surface as HTTP 500 rather than proper 422 validation errors.

So the specific Pi + Claude Haiku + subscription auth combo appears to be one that falls through the existing frontend harness-capability filter. The fix likely involves two layers:

1. Frontend: ensuring the /inspect-driven filter correctly excludes Claude Haiku under the Pi harness (or flags it with a warning).
2. Backend: reviving the save-time validation from PR feat(api): validate the agent template on commit and test_run #5104 as a safety net, so invalid combos never persist silently.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.