glm5_next W9c-2: lift the RouteTopk and KDA CPU-only refusals - #3133
Merged
localai-org-maint-bot merged 1 commit intoSep 12, 2026
Merged
Conversation
mudler
force-pushed
the
row/MODEL-MM-GLM53-FLASH-W9C2-MOE-DEV
branch
2 times, most recently
from
September 12, 2026 00:32
456d961 to
d41567f
Compare
This was referenced Sep 12, 2026
Open
Merged
localai-org-maint-bot
added a commit
that referenced
this pull request
Sep 12, 2026
…h the ViewOn contract (#3154) test_rocm_f16_contract (introduced in #3095) failed on clean main because its check lambda expected `OwnedTensor::View()` and `ViewOn()` to propagate the layout markers `repacked`, `q8_0_aligned`, and `elem_kn_repacked`. `ViewOn` deliberately carries only `weight_value_dtype`: each `ResidentWeight` arm owns a different marker set and sets them explicitly after the call, and copying them centrally changed the CPU-alias lane for every model that shares the helper. The test and the code contradicted each other in the same commit, and `build-test-cpu` was skipping when #3095 merged so the failure was never caught. The fix corrects the check lambda to verify the actual contract: `weight_value_dtype` IS propagated, layout markers are NOT. The `Slice` guard is removed because `View()` no longer propagates `repacked`, so `Slice()` never throws on a view from an `OwnedTensor` that happens to have `repacked` set. This unblocks every PR that runs `build-test-cpu` (#3133, #3140, #2906, and others). Closes #3153. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo/glm5.2 [maki] Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
…ls (W9c-2) RouteTopk and Glm5NextKdaLayerForward each refused any non-CPU queue by name because vt::MoeRouterTopK and vt::KdaGatedDeltaRule dispatch on the queue's device, and handing them host pointers on a GPU queue is a crash, not a fallback. The device arm was deferred to the assembled text forward (W9c-3), so the refusal prevented the KDA and MoE layers from reaching the CUDA and ROCm providers they already call. This change adds a `dense_attn::Dev* dev = nullptr` parameter to both functions. When `dev` is non-null the routing and recurrence operands are uploaded via `dense_attn::DBuf`, the op runs on `dev->q`, and the outputs are downloaded back to the host vectors. The projections, conv, gates, norms and the host arm's original error path are unchanged. When `dev` is null the host arm runs exactly as before and still refuses a non-CPU queue by name. DecoderLayerForward threads `dev` into the KDA call site; the MoeForward call site already carried `dev` from the prior placement work (PR #3128). The KDA device-arm test iterates over CUDA and ROCm backends (mirroring the k-pool device test's KpoolDeviceTypes pattern), runs the device arm against the host arm oracle, and asserts a non-zero NMSE band. On a CPU-only build it skips loudly. The existing MoE CUDA device-arm test (W9c-3a) already exercises RouteTopk's new device arm through MoeForward. Issue: #2410 (W9c-2 wave) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:regolo-glm5.2 [maki]
mudler
force-pushed
the
row/MODEL-MM-GLM53-FLASH-W9C2-MOE-DEV
branch
from
September 12, 2026 07:15
d41567f to
3cd917b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
glm5_next W9c-2: lift the RouteTopk and KDA CPU-only refusals
RouteTopk and Glm5NextKdaLayerForward each refused any non-CPU queue
by name because vt::MoeRouterTopK and vt::KdaGatedDeltaRule dispatch on
the queue's device, and handing them host pointers on a GPU queue is a
crash, not a fallback. The device arm was deferred to the assembled
text forward (W9c-3), so the refusal prevented the KDA and MoE layers
from reaching the CUDA and ROCm providers they already call.
This change adds a
dense_attn::Dev* dev = nullptrparameter to bothfunctions. When
devis non-null the routing and recurrence operandsare uploaded via
dense_attn::DBuf, the op runs ondev->q, and theoutputs are downloaded back to the host vectors. The projections, conv,
gates, norms and the host arm's original error path are unchanged. When
devis null the host arm runs exactly as before and still refuses anon-CPU queue by name.
DecoderLayerForward threads
devinto the KDA call site; the MoeForwardcall site already carried
devfrom the prior placement work (PR #3128).The KDA device-arm test iterates over CUDA and ROCm backends (mirroring
the k-pool device test's KpoolDeviceTypes pattern), runs the device arm
against the host arm oracle, and asserts a non-zero NMSE band. On a
CPU-only build it skips loudly. The existing MoE CUDA device-arm test
(W9c-3a) already exercises RouteTopk's new device arm through MoeForward.
Issue: #2410 (W9c-2 wave)
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo-glm5.2 [maki]