Qwen3.6-27B: no working mlx-lm version exists — qwen3_5 unsupported on 0.30.0, Stream(gpu,3) crash on 0.31.3MLX-LM Mismatch

[ltrudeau-design]

Rapid-MLX version

0.6.1

Hardware

MacBook Pro M1, 64GB unified memory

macOS version

Sequoia 15.0

Python version

3.14

Model

Qwen3.6-27B 4-bit (local path: ~/.cluster/MLX_Models/qwen36-27b-local)

Full serve command

rapid-mlx serve ~/.cluster/MLX_Models/qwen36-27b-local --host 127.0.0.1 --port 8081 --max-tokens 2048 --prefill-step-size 2048 --no-thinking

What happened?

Package requires mlx-lm>=0.31.0 but no version in that range works with Qwen3.6-27B:

mlx-lm 0.30.0 → ValueError: Model type qwen3_5 not supported
mlx-lm 0.31.3 → RuntimeError: There is no Stream(gpu, 3) in current thread — server starts, accepts requests, but returns 1 token in 21s at 0.0 tok/s

The 0.31+ crash appears to be caused by the removal of the internal Batch API in mlx-lm 0.31+. The log shows: [chunked_prefill] Skipped — mlx-lm 0.31+ removed the internal Batch API.
Qwen3.6 was advertised as Day 0 supported in the v0.6.1 release notes.

Steps to reproduce

Create a fresh venv: python3 -m venv ~/.mlx-venv
Activate: source ~/.mlx-venv/bin/activate
Install: pip install rapid-mlx (resolves mlx-lm 0.31.3)
Serve any Qwen3.6-27B 4-bit model: rapid-mlx serve --port 8081
Send any chat completion request to http://127.0.0.1:8081/v1/chat/completions
Observe: server accepts request, runs for ~21s, returns 1 token at 0.0 tok/s, logs RuntimeError: There is no Stream(gpu, 3) in current thread

To confirm the version conflict:

Downgrade: pip install "mlx-lm==0.30.0"
Restart server — now fails at startup with ValueError: Model type qwen3_5 not supported

No mlx-lm version satisfies both the model architecture requirement and the batched engine.

Error logs / output

[raullenchai]

Thanks for the very precise repro @ltrudeau-design — this is already fixed on main as of two days ago.

What you hit

The RuntimeError: There is no Stream(gpu, 3) in current thread is the same root cause as #160. mlx-lm 0.31+ now requires the MLX generation stream to be created on the same OS thread that runs generate_step. Our BatchedEngine runs that on a dedicated worker thread (so the FastAPI event loop never blocks), and v0.6.1 wasn't initializing the stream on that worker. Result: every generation call hit the wrong-thread guard, the worker swallowed the error and returned 1 token after the first stop check.

Fix

PR #161 (88eadd8, merged 2026-04-25) attaches mx.new_stream(mx.default_device()) inside the worker thread initializer so mlx_lm.generate.generation_stream resolves to the correct stream. Your trace and the [chunked_prefill] Skipped — mlx-lm 0.31+ removed the internal Batch API log line are both consistent with that path.

Workaround until 0.6.2

Until the release lands, install from main:

pip install git+https://github.com/raullenchai/Rapid-MLX.git@main

Then re-run your exact command — the Stream(gpu, 3) error should disappear and you should see normal tok/s for Qwen3.6-27B 4-bit on M1.

0.6.2 release

Cutting 0.6.2 today. It bundles four fixes that all touch this surface:

• Fix MLX generation stream thread ownership #161 — MLX generation stream thread ownership (your crash)
• prefix cache stores entries but never serves them for hybrid models #163 — hybrid prefix cache MISS forever on mlx-lm 0.31+
• cleanup: tidy _init_mlx_step_thread (follow-up to #161) #164 — cleanup of Fix MLX generation stream thread ownership #161
• fix: restore prompt-boundary cache snapshot for mlx-lm 0.31+ (#163) #165 — thread-safe MemoryAwarePrefixCache (race exposed by prefix cache stores entries but never serves them for hybrid models #163)

Will close this issue once 0.6.2 is on PyPI. If the main install doesn't unblock you, please reopen with the new logs.

[raullenchai]

v0.6.2 is out: https://github.com/raullenchai/Rapid-MLX/releases/tag/v0.6.2

pip install -U rapid-mlx

Closing as fixed. Please reopen with new logs if the Stream(gpu, 3) error persists for you on 0.6.2.

[raullenchai]

Quick follow-up: while doing onboarding QA on the freshly-released 0.6.2, we hit a smaller secondary instance of the same bug — the prefix-cache persistence at shutdown was still touching MLX arrays from the asyncio thread, so ~5/119 entries silently dropped on every restart plus a trailing RuntimeError traceback in logs (different code path from your generation crash).

Fixed in v0.6.3 (just published, #167):

• 119/119 cache entries persist on shutdown (was 114/119 on 0.6.2)
• Zero Stream(gpu, N) warnings
• Zero trailing tracebacks

Verified by re-running your exact command against fresh pip install rapid-mlx==0.6.3 and brew upgrade rapid-mlx — both clean.

pip install -U rapid-mlx

Release notes: https://github.com/raullenchai/Rapid-MLX/releases/tag/v0.6.3

Leaving this closed; thanks again for the precise repro that surfaced the whole chain.

[raullenchai]

Thanks @ltrudeau-design for opening this — feedback from people using and building
with rapid-mlx helps shape what we improve next. 🙏

Beyond this issue, what's the roughest edge you've hit with rapid-mlx?
Reply here, or drop it in our feedback thread →
#1205

Also curious: what are you using rapid-mlx for, and what does your setup
look like? Even a couple of lines would help us prioritize the right things.

If you'd rather chat 1:1, happy to connect by email or whatever channel
is easiest. And if you ever write up your setup (blog/gist/tweet), ping me
and I'll amplify it.