Skip to content

fix(shortlist): drop option_order when a narrow no longer fits it - #1063

Open
Bruce-Yii wants to merge 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:fix/shortlist-drop-option-order-on-narrow
Open

Bruce-Yii wants to merge 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:fix/shortlist-drop-option-order-on-narrow

Conversation

@Bruce-Yii

Copy link
Copy Markdown
Contributor

What

predict_shortlist narrows a choice's criteria to the top-k labels while forwarding the caller's full-length option_order, and predict_tournament builds its group and finalist questions the same way in cut(). The narrowed question then fails Agent._check_question, because the order must be a permutation of range(n_options) and the kept label set is smaller. Reproduced on main 1adc59f7:

ValueError: question 'pick': 'option_order' must be a permutation of range(2) -- one slot per option, each option once -- got [5, 4, 3, 2, 1, 0]

Fix

Following the direction you gave on #958, a narrowed choice now drops its option_order with one RuntimeWarning naming the question, the old slot count and the kept label count. The order is not remapped onto the kept labels, since that would silently move options between slots.

Both paths are covered: predict_shortlist, and cut() in predict_tournament (the round groups and the finalists share that choke point, so one helper covers them). The warning fires once per question per call, not once per tournament round.

Unchanged: passthrough questions (n <= k), non-choice questions, the caller's own dicts (both paths copy before editing), and calls without an option_order — those emit no warning and behave as before, so a call with an order answers exactly like the same call without one. Verified on real weights for both paths.

Tests

  • python tests/test_shortlist.py → 219 passed, 0 failed (on main: 209 passed, 10 failed — the 10 new checks)
  • python tests/test_hooks_api.py → 572 passed, 0 failed
  • real weights (local english checkpoint, CPU): both paths answer identically with and without the order; one warning per narrowed question; passthrough keeps its order
  • repo gates: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean; python -m compileall -q laya/ tests/ exit 0
  • tests/test_mcp.py could not run in my environment: the installed mcp is 1.29.1 and the suite requires >= 2.2.0. It fails the same way on unmodified main, so this is an environment block, not a result — and MCP behaviour is unchanged, since fix(mcp): keep and validate a question's option_order #958 already makes laya_shortlist refuse a narrowed question that carries an order.

Two commits: the regression tests first (red on main, green with the fix), then the fix itself.

Closes #1062

A choice narrowed by predict_shortlist or predict_tournament kept the caller's full-length option_order, which Agent._check_question then refuses: the order must be a permutation of range(n_options) and the kept label set is smaller. Nandha called this a core contract question on NandhaKishorM#958 and asked for the order to be dropped with a warning. These checks are red on main; they pin the narrowed question validating, one RuntimeWarning naming the question, an answer identical to the no-order call, and passthrough/non-choice questions untouched.
predict_shortlist narrowed criteria to the top-k labels while forwarding the caller's full-length option_order, and predict_tournament built its group and finalist questions the same way, so a choice carrying an order crashed in the agent: the order must be a permutation of range(n_options) and the kept label set is smaller. Nandha called this a core contract question on NandhaKishorM#958 and asked for the order to be dropped with a warning, since remapping it onto the kept labels would silently move options between slots.

Both paths now drop the order on a narrowed question and emit one RuntimeWarning naming the question and the slot counts. Passthrough questions, non-choice questions and the caller's own dicts are untouched, so a call with an order answers exactly like the same call without one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: option_order becomes invalid when shortlist or tournament narrows a choice

1 participant