feat: add local Qwen3-Omni API and fine-tune regression

[glennko]

• swap Qwen3-Omni dataset generation to load Hugging Face checkpoints locally and wire up tests/ docs
• ensure AGENTS.md/CLAUDE.md stay untracked
• add a lightweight GPT-2 fine-tuning test that asserts weights actually change

Summary

Checklist

• Tested
• Documented

Additional Information

[MarcosRiveraMartinez]

It doesn't look correct. I don't think AutoModelForCausalLM is supported for this model. Just try AutoModelForMultimodalLM

[glennko]

I've made the suggested changes.

[glennko]

Review findings

Re-reviewed against current main. The approval here predates ~7 months of drift, so flagging what's changed and what I think blocks.

The rebase is a non-issue. Merge base is fb16cc2; every commit main has gained since is a Dependabot bump to docs/package-lock.json plus the lockfile refresh in #342. Zero overlap with this PR's changed files — BEHIND here is pure branch-protection bookkeeping, not conceptual drift. CI gates pass on the head: black --check clean, isort clean, scripts/check_docs_contracts.py passes.

Blocking

1. The API is unreachable on a supported install. qwen.py needs AutoModelForMultimodalLM, which requires transformers>=5.0.0, but pyproject.toml:46 still pins >=4.36.0 and this PR doesn't bump it. Confirmed against the resolved 4.36.2: hasattr(transformers, "AutoModelForMultimodalLM") is False. So every construction of Qwen3OmniTextGenerationAPI raises the ImportError at qwen.py:76 telling the user to upgrade — and that upgrade would likely break the rest of xTuring, which is written against 4.x. Meanwhile the README and docs/docs/advanced/generate.md advertise it as usable. Needs a pin bump, an optional extra, or a feature gate.

Also worth double-checking that AutoModelForMultimodalLM is the real symbol — Qwen3-Omni's own class is Qwen3OmniMoeForConditionalGeneration, and nothing in the repo can validate the name.

2. finish_reason is hardcoded, which silently corrupts the pipeline this exists to feed. qwen.py:129 always emits "stop", even when generation stopped on max_new_tokens. The self-instruct pipeline discards truncated completions by branching on finish_reason == "length" — see bootstrap_instructions.py:71 and prepare_for_finetuning.py:192,235. With this engine that branch is dead, so truncated mid-sentence generations get accepted into the generated dataset. Should compare completion token count against max_tokens and emit "length".

3. The "fine-tune regression" in the title never runs. Two independent gates: .github/workflows/ci.yml uses an explicit allowlist (tests/xturing/cli/test_api_server.py, tests/xturing/evaluation/test_runner.py) that includes neither new test file; and test_gpt2_model.py:24 gates on XTURING_RUN_TRAINING_TESTS == "1", which is set nowhere in the repo. As written it's documentation, not a regression guard.

The bigger miss is the 6 Qwen tests — they're mocked, CPU-only, fast, dependency-light, and would be a genuinely cheap CI addition. #343 adds a model-api-tests job covering tests/xturing/model_apis/, which would pick them up; worth landing that first and confirming these run under it.

4. Pad tokens aren't masked in the PEFT example. train_qwen3_omni_peft.py:229-233 clones input_ids into labels and masks only the prompt prefix; padding positions stay unmasked, so with padding=True and batch>1 the model computes loss over pad tokens. The sibling pad_batch() at line 62 does this correctly, so it's an inconsistency rather than a deliberate choice. Related: prompt_lens = prompt_inputs["attention_mask"].sum(dim=1) is only correct under right-padding.

Non-blocking

• train_qwen3_omni_peft.py:243 passes tokenizer= to Trainer (deprecated 4.46, removed in 5.0.0 — the exact version this file needs) and torch_dtype= at line 141 (renamed to dtype=). Nothing catches it because the file is never imported.
• README.md:181 gained a stray leading space inside a Python fence — copy-pasting now yields IndentationError.
• The new "Running Tests" block claims @pytest.mark.slow/@pytest.mark.gpu are "used in this project". pytest.ini:4-7 registers them but no test uses either, so pytest -m "not slow" deselects nothing.
• "Quick Contribution Guidelines" says PRs should target dev, contradicting CONTRIBUTING.md:44 ("target main") — and this PR itself.
• docs/docs/advanced/generate.md links to Qwen2.5-Omni and uses model_name_or_path="Qwen/Qwen2.5-Omni" (not a valid repo id) in a Qwen3-Omni section.
• .gitignore now ignores AGENTS.md/CLAUDE.md repo-wide, so nobody can ever commit those filenames. Note docs(contributing): drop dead AGENTS.md link #344 removes a CONTRIBUTING.md link to AGENTS.md.

Verified fine

generate_text correctly absorbs and discards the OpenAI-only kwargs (frequency_penalty, presence_penalty, logprobs, best_of) that all three self-instruct call sites pass, so they don't leak into model.generate(). The hard import torch in model_apis/qwen.py:5 does not break lightweight-tests — that job installs only pytest fastapi uvicorn httpx, and neither xturing.cli nor xturing.evaluation imports model_apis. Test quality is good: finetuning_config() returns the live args object, distilgpt2 defaults guarantee 2 optimizer steps fire, and weight_shift > 0.0 isn't flaky.

[glennko]

Parking this — and a correction in your favour

Researched the transformers 5.x question properly, verified against the v5.16.1 source tree rather than from memory. Two of my earlier points need revising.

Correction: AutoModelForMultimodalLM is real

I said it was worth double-checking whether that symbol exists. It does. It's defined at models/auto/modeling_auto.py:2439 in v5.x, exported in __all__, and documented in model_doc/auto.md. It's new in v5.0.0 — grepping v4.56 and v4.57 returns zero hits, which is why it looked suspicious.

Your class mapping is also correct: MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES maps qwen3_omni_moe → Qwen3OmniMoeForConditionalGeneration, so AutoModelForMultimodalLM is the right auto-class, and it's the broadest any-to-any one. Qwen3-Omni support itself landed in v4.57.0, but only under AutoModelForTextToWaveform; the AutoModelForMultimodalLM route genuinely is v5-only. The approach in this PR is sound.

Confirmed: the pin bump is breaking, and worse than the README said

load_in_8bit= / load_in_4bit= were hard-removed from from_pretrained in v5.0.0 (PR #41287), not deprecated. Zero hits in modeling_utils.py at v5.16.1. The failure mode is a bare TypeError: __init__() got an unexpected keyword argument 'load_in_8bit' — no helpful migration error.

xturing passes them as bare kwargs in 7 places: engines/causal.py (50, 94, 251), llama_engine.py (60, 86), gptj_engine.py (46, 66). That part is a mechanical fix — BitsAndBytesConfig(load_in_8bit=True) passed as quantization_config=, and causal.py:83 already does exactly this for the woq path, so the pattern is in the codebase.

But the quantization kwargs are the small problem. Three pins in pyproject.toml sit below v5's hard floors:

declared	v5.x requires
bitsandbytes==0.41.1 (exact pin)	>= 0.46.1
accelerate==0.22.0 (exact pin)	>= 1.1.0
torch >= 1.9.0	>= 2.5
requires-python = ">=3.7"	>= 3.10

Plus a long tail that a LoRA library walks straight into: Trainer(tokenizer=) removed in favour of processing_class= (no **kwargs, so it's an immediate TypeError), TrainingArguments.warmup_ratio removed, safe_serialization=False removed, default dtype changed from fp32 to "auto", transformers.tokenization_utils moved (you import from it in 4 files — 3 collators plus llama_utils/llama.py), and use_auth_token → token (2 sites in quant_utils/). MoE experts were also refactored off nn.ModuleList, which broke PEFT adapters on MoE models — and Qwen3-Omni is MoE, so that's directly relevant here. PEFT floor for v5 is >= 0.19.1.

Why it's parked

This isn't a "fix four review comments" PR. Landing Qwen3-Omni means a transformers 5.x migration across the whole library — a dependency-floor bump that drops Python 3.7–3.9, plus quantization, Trainer, and tokenizer-import changes. That's its own epic, and it should not ride in on a feature PR.

The other three findings from my earlier review still stand and are worth fixing whenever this resumes — finish_reason hardcoded to "stop" at qwen.py:129 is the one with real consequences, since it silently feeds truncated generations into the self-instruct dataset.

#358 corrects the README note that #320 added, so main no longer points people at a 5.x upgrade that would break their INT8/INT4 install.