RuntimeError: There is no Stream(gpu, 0) in current thread.

[mlaprise]

Rapid-MLX version

rapid-mlx 0.6.1

Hardware

MacBook Pro M5 Max 48gb

macOS version

26.4.1

Python version

3.13.13

Model

qwen3.6-35b

Full serve command

rapid-mlx serve qwen3.6-35b

What happened?

Latest version don't run on my M5 Max

Steps to reproduce

1. git clone ...
2. uv sync
3. rapid-mlx serve qwen3.6-35b

Error logs / output

Traceback (most recent call last):
  File "/Users/mlaprise/repos/Rapid-MLX/vllm_mlx/scheduler.py", line 2343, in step
    raw_next = self.batch_generator.next()
  File "/Users/mlaprise/repos/Rapid-MLX/.venv/lib/python3.13/site-packages/mlx_lm/generate.py", line 1329, in next
    return self._next()
           ~~~~~~~~~~^^
  File "/Users/mlaprise/repos/Rapid-MLX/vllm_mlx/scheduler.py", line 564, in _chunked_next
    new_batch = self._process_prompts(batch_prompts)
  File "/Users/mlaprise/repos/Rapid-MLX/vllm_mlx/scheduler.py", line 236, in _patched_process_prompts
    batch = _orig_process_prompts(prompts)
  File "/Users/mlaprise/repos/Rapid-MLX/.venv/lib/python3.13/site-packages/mlx_lm/generate.py", line 1107, in _process_prompts
    mx.eval([c.state for c in prompt_cache])
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: There is no Stream(gpu, 0) in current thread.

[mostalive]

I'm seeing the same with qwen3.6:27 b, M3 Max 64GB macbook pro:

 rapid-mlx serve --prefill-step-size 8192 --port 8000 qwen3.6-27b
  Alias: qwen3.6-27b → mlx-community/Qwen3.6-27B-4bit
INFO:vllm_mlx.model_auto_config:Auto-detected model family 'qwen3\.6' → tool_call_parser=qwen3_coder_xml, reasoning_parser=qwen3
INFO:vllm_mlx.cli:Auto-configured --tool-call-parser qwen3_coder_xml
INFO:vllm_mlx.cli:Auto-configured --reasoning-parser qwen3
INFO:vllm_mlx.cli:Reasoning parser enabled: qwen3

  Rapid-MLX
  ─────────
  Features: tools: qwen3_coder_xml, reasoning: qwen3, gc-control
  Model: mlx-community/Qwen3.6-27B-4bit
Mode: Continuous batching (for multiple concurrent users)
Stream interval: 1 tokens
Memory-aware cache: 20% of RAM
INFO:vllm_mlx.server:Loading model with BatchedEngine: mlx-community/Qwen3.6-27B-4bit
INFO:vllm_mlx.server:Model loaded: mlx-community/Qwen3.6-27B-4bit
INFO:vllm_mlx.server:Default max tokens: 32768
INFO:vllm_mlx.runtime.model_registry:Registered model 'mlx-community/Qwen3.6-27B-4bit' (aliases={'qwen3.6-27b'}, default=True)

  Ready: http://localhost:8000/v1
  Docs:  http://localhost:8000/docs

INFO:     Started server process [90177]
INFO:     Waiting for application startup.
INFO:vllm_mlx.server:GC control enabled: thresholds set to (100000, 50, 50)
INFO:httpx:HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3.6-27B-4bit/revision/main "HTTP/1.1 200 OK"
Fetching 16 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 140395.11it/s]
Download complete: : 0.00B [00:00, ?B/s]              INFO:httpx:HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3.6-27B-4bit/revision/main "HTTP/1.1 200 OK"                      | 0/16 [00:00<?, ?it/s]
Fetching 14 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 103746.04it/s]
Download complete: : 0.00B [00:00, ?B/s] [00:00, ?B/s]
Download complete: : 0.00B [00:00, ?B/s]                                                                                                                                                                 | 0/14 [00:00<?, ?it/s]
INFO:httpx:HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3.6-27B-4bit/revision/main "HTTP/1.1 200 OK"
Fetching 14 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 113578.83it/s]
Download complete: : 0.00B [00:00, ?B/s]              INFO:vllm_mlx.engine.batched:Metal memory limits set: allocation_limit=50.1GB (90% of 55.7GB), cache_limit=32GB                                    | 0/14 [00:00<?, ?it/s]
INFO:vllm_mlx.memory_cache:MemoryAwarePrefixCache initialized: max_memory=7325.2MB, max_entries=1000
INFO:vllm_mlx.scheduler:Memory-aware cache enabled: limit=7325.2MB
INFO:vllm_mlx.engine_core:Hybrid model detected (GatedDeltaNet) — enabling 200ms request throttle for batch safety
INFO:vllm_mlx.engine_core:Engine started
INFO:vllm_mlx.engine.batched:BatchedEngine loaded: mlx-community/Qwen3.6-27B-4bit (mllm=False)
INFO:vllm_mlx.server:Warming up (compiling Metal shaders)...
INFO:vllm_mlx.server:Hybrid model: running full request warmup (compiling GatedDeltaNet kernels)
INFO:vllm_mlx.scheduler:[cache_fetch] request=8a57e9c9-8cb MISS prompt_tokens=11 time=0.000s entries=0
INFO:vllm_mlx.engine_core:[stream_outputs] 8a57e9c9-8cb START waiting for tokens
WARNING:vllm_mlx.scheduler:[chunked_prefill] Skipped — mlx-lm 0.31+ removed the internal Batch API. Using native prefill_step_size=2048 instead. Memory-aware cache and mid-prefill snapshots are unavailable.
INFO:vllm_mlx.scheduler:[schedule] request=8a57e9c9-8cb uid=0 prompt_tokens=11 tokens_to_prefill=11 max_tokens=2 running=1 waiting=0
ERROR:vllm_mlx.scheduler:Error in batch generation step: There is no Stream(gpu, 3) in current thread.
Traceback (most recent call last):
  File "/Users/willem/.local/share/uv/tools/rapid-mlx/lib/python3.12/site-packages/vllm_mlx/scheduler.py", line 2343, in step
    raw_next = self.batch_generator.next()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/willem/.local/share/uv/tools/rapid-mlx/lib/python3.12/site-packages/mlx_lm/generate.py", line 1855, in next
    return self._next()
           ^^^^^^^^^^^^
  File "/Users/willem/.local/share/uv/tools/rapid-mlx/lib/python3.12/site-packages/mlx_lm/generate.py", line 1809, in _next
    gen_batch = self._prompt_batch.split(split).generate(last_inputs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/willem/.local/share/uv/tools/rapid-mlx/lib/python3.12/site-packages/mlx_lm/generate.py", line 1186, in generate
    generation = GenerationBatch(
                 ^^^^^^^^^^^^^^^^
  File "/Users/willem/.local/share/uv/tools/rapid-mlx/lib/python3.12/site-packages/mlx_lm/generate.py", line 1288, in __init__
    self._step()
  File "/Users/willem/.local/share/uv/tools/rapid-mlx/lib/python3.12/site-packages/mlx_lm/generate.py", line 1369, in _step
    mx.async_eval(self._next_tokens, self._next_logprobs, token_context)
RuntimeError: There is no Stream(gpu, 3) in current thread.

[dougborg]

Reproducing on M1 Max 64GB / rapid-mlx 0.6.1 / mlx-lm 0.31.3 — adding a sequential-request pattern that might help narrow the cause, plus a masking issue worth surfacing separately.

Pattern: only the second-and-subsequent requests fail

Running 4 OpenAI-shaped /v1/chat/completions calls with tools sequentially against the same server, all with mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (qwen3_coder parser):

#	scenario	calls	latency	result
1	get_weather(Tokyo)	1/1	2086 ms	✅ real response
2	no-tool conversational	0/0	39 ms	✅ (empty response, trivially passes)
3	calculator(47,83,multiply)	0/1	36 ms	❌ Stream(gpu, 4)
4	parallel get_weather ×2	0/2	36 ms	❌ Stream(gpu, 4)

First request always succeeds. Subsequent requests fail in ~40ms, well before any model work. Persists across rm -rf ~/.cache/vllm-mlx/prefix_cache.

Server-log correlation

Every failing request shows a [cache_fetch] HIT immediately before the Stream error:

INFO:vllm_mlx.scheduler:[cache_fetch] request=... HIT prompt_tokens=401 cached=378 remaining=23
INFO:vllm_mlx.scheduler:[schedule] request=... prompt_tokens=401 tokens_to_prefill=23, 378 cached
ERROR:vllm_mlx.scheduler:Error in batch generation step: There is no Stream(gpu, 4) in current thread.
  ... mx.eval([c.state for c in self.prompt_cache])

Suggests cache entries from request 1's thread-local GPU stream are referenced from request 2's thread and the stream is gone. Consistent with the removed-internals note in #117.

Separate issue: silent 200 OK on stream failure

The error is caught by [generation_error_recovery] aborted 1 running requests, batch generator closed, Metal cache cleared and the HTTP response is 200 OK with 0 completion tokens rather than a 5xx. Clients can't distinguish "model had nothing to say" from "scheduler blew up". Would be worth returning an error status (or at least a non-empty finish_reason: "error") so downstream tooling sees it.

Hardware coverage so far

M5 Max 48GB (OP), M3 Max 64GB (mostalive), M1 Max 64GB (this comment) — spans three Apple Silicon generations.

No working rapid-mlx version on mlx-lm 0.31.3 (correction)

Pinning to rapid-mlx 0.5.10 (pre-0.6.x) on our end as a stopgap.

Update: verified the bug reproduces identically on rapid-mlx 0.5.10 with the same repro pattern and stack trace. Both 0.5.10 and 0.6.1 resolve to mlx-lm 0.31.3 in a fresh uv tool install env, so the trigger is the mlx-lm 0.31 batch-API rewrite breaking the scheduler monkey-patches (as #117 calls out). No known-good rapid-mlx version on current mlx-lm — downgrading rapid-mlx alone doesn't help.

Happy to share the bench-tools script we used if a minimal repro helps — it's just 4 sequential httpx POSTs with a tools array, nothing fancy.

[samuelfaj]

Same problem here.

Mac m5 max.

[westNeighbor]

was working well, moments ago I upgrade to the latest and I met the same problem here
m2 max

ERROR:vllm_mlx.scheduler:Error in batch generation step: There is no Stream(gpu, 3) in current thread.
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/rapid-mlx/0.6.1/libexec/lib/python3.12/site-packages/vllm_mlx/scheduler.py", line 2343, in step
    raw_next = self.batch_generator.next()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/rapid-mlx/0.6.1/libexec/lib/python3.12/site-packages/mlx_lm/generate.py", line 1855, in next
    return self._next()
           ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/rapid-mlx/0.6.1/libexec/lib/python3.12/site-packages/mlx_lm/generate.py", line 1841, in _next
    self._prompt_batch.prompt(prompts)
  File "/opt/homebrew/Cellar/rapid-mlx/0.6.1/libexec/lib/python3.12/site-packages/mlx_lm/generate.py", line 1161, in prompt
    mx.eval([c.state for c in self.prompt_cache])
RuntimeError: There is no Stream(gpu, 3) in current thread.

WARNING:vllm_mlx.scheduler:[generation_error_recovery] aborted 1 running requests, batch generator closed, Metal cache cleared

[samuelfaj]

Fix: #161

Fix for Qwen 3.6 api translations: #162