tenstorrent: decode IQ3_XXS matmul weights on the int8-dot kernel (QUANT-GGUF-IQ-TENSTORRENT wave 1) - #3193
Merged
Conversation
… int8-dot arm Committed before implementation. Scope, upstream anchors (CPU VecDotIQ3_XXSQ8_K oracle, block layout, tables, route predicate, gate artifact pin with sha256), design (device dot, staging wpb=32, enc_sel=4, default-path dispatch, route predicate + pin test in the same change), risks, red-first tests, gates, owed items, stop conditions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
QUANT-GGUF-IQ-TENSTORRENT wave 1. The keep-quant set on the P150 covered Q4_K/Q5_K/Q6_K/Q8_0, so an IQ3_XXS weight refused at first matmul forward and the mudler/Qwen3.8-27B-APEX-GGUF I-Nano artifact (164 IQ3_XXS tensors of 506) could not load on the device. Red first: the int8-dot sweep gained an IQ3_XXSxq8_K case and the TENSTORRENT route pin gained an admission CHECK; both ran red against the unfixed tree (route refusal at the registered-set VT_CHECK, and kExpandBf16 from the predicate). Evidence under /tmp/row-tt-iq-evidence. The fix adds kq_vec_dot_iq3_xxs_q8_K to the shared device math header, the packed-stream port of the CPU VecDotIQ3_XXSQ8_K reference (quants.c:999): per 32-element sub-block a scale+sign u32 (ls = 2*(aux32>>28)+1), four kIq3xxsGrid lookups with kKsignsIq2xs signs, int32 accumulation in the CPU's order, one d*bsum fold per block and the 0.25f grid-magnitude fold last. No divisions, so the file's reciprocal hazard does not arise. The three codebook tables move into src/vt/tenstorrent/kernels/iq3xxs_tables.h because the device include path is the kernels dir only; the CPU header keeps its own copy and the bit-exact op-level sweep is the drift guard in both directions. Staging is 32 words per block (98 B zero-padded to 128 B, the Q6_K sub-word-tail pattern; EnsureKeepQuantWords already zero-fills the pad and the decoder never reads past byte 98). The kernel dispatches enc_sel 4 to the new dot with the existing q8_K activation quantizer, and MatmulBTQuantKernel routes IQ3_XXS to the int8-dot arm REGARDLESS of VT_TT_KEEPQUANT_INT8DOT, because the grouped arm has no IQ3_XXS decode to fall through to and the capability must be reachable on the default configuration. DeviceKeepQuantSupported admits kIQ3_XXS on kTENSTORRENT in the same change, and docs/USAGE.md records the gated artifact with its repository, revision, size and sha256, the refused arms (IQ2_S/IQ2_XXS/Q3_K) named beside it. Green: op-level sweep bit-exact vs the CPU oracle across the 18-shape bf16-act sweep plus the f32-act leg (enc=16 included); route pin green; full backend suite 73/73; full test_gguf_keep_quant 61/61; vllm-bench links. The APEX e2e vllm-bench gate remains owed to the row spec (Gates 4). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
The wave-1 commit's headline claim — IQ3_XXS dispatches to the int8-dot kernel REGARDLESS of VT_TT_KEEPQUANT_INT8DOT, which is what makes the capability reachable on the default configuration (tenstorrent_ops.cpp MatmulBTQuantKernel) — was not pinned by any test. The int8-dot sweep self-gates on the env (it skips loudly unless VT_TT_KEEPQUANT_INT8DOT=1), and the route pin proves admission, not dispatch: deleting the dispatch override ALONE left every gate green while the default path fell through to the W4a grouped arm that refuses kIq3_xxs outright. This adds a default-path leg to the keep-quant op suite: a dedicated TEST_CASE that unsets VT_TT_KEEPQUANT_INT8DOT (restoring the ambient value on exit, the microbench's save/restore pattern) and runs the IQ3_XXS device decode across 8 shapes at the same bit-exact bar vs the CPU integer vec_dot oracle the sweep holds. Verified red-first with the reviewer's exact mutation — the dispatch override deleted, admission kept — which throws the kIq3_xxs grouped-route refusal here, and green again once the override is restored. Full backend suite: 74/74. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…X e2e recorded OOM/owed The op-level oracle, route admission, and default-path dispatch are green (74/74 backend suite, review mutations verified); the APEX-I-Nano e2e generation gate fatals on the PRE-EXISTING Q4_K grouped decode's kq-decode/repair plane (1.117 GB ask, 108 MB largest free block — APEX's refused-arm bf16 expansions fragment the allocator), which is keep-quant W4 residency territory, not an IQ3_XXS defect. Evidence: /tmp/row-tt-iq-evidence/apex-alloc-trace.log. Spec ## Now/## Evidence/## Owed, the tracker issue's Resolution, and the quantization-matrix state (INVENTORIED -> PARTIAL) all carry the dated record in the same change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave 1 of QUANT-GGUF-IQ-TENSTORRENT:
DType::kIQ3_XXSmatmul weights decode on-core through the TT int8-dot keep-quant kernel, bit-exact vs the CPU oracle, with loader route admission on the default path (no env). Spec committed first (69e0263), implementation reviewed by mutation (4/5 guarantees red-detected; the dispatch-only gap the review found is repaired by a dedicated env-unset pin leg).What lands
kq_vec_dot_iq3_xxs_q8_Kinkeepquant_kernel_code.h— the CPUVecDotIQ3_XXSQ8_Kport (cpu_quant_dot.cpp:622, quants.c:999), int32 accumulation in the ported order, no divisions; codebook tables in a newkernels/iq3xxs_tables.h(device include path; the oracle test is the drift guard vs the CPU header).wpb=32(98 B zero-padded to 128 B — the Q6_K sub-word-tail pattern),enc_sel=4, kernelenc==4arm; activation side takes the existing q8_K quantizer.MatmulBTQuantKernelroutes IQ3_XXS to the int8-dot arm REGARDLESS ofVT_TT_KEEPQUANT_INT8DOT(default-path reachability); refusal message names the encoding.DeviceKeepQuantSupportedkTENSTORRENT admits kIQ3_XXS; thetest_gguf_keep_quant.cpppin widens in the same change (the predicate file's own rule).docs/USAGE.mdrecords the gate artifact:mudler/Qwen3.8-27B-APEX-GGUF@98454f31de8ac2e8bc7cd359c526d9db230ca547,Qwen3.8-27B-APEX-I-Nano.gguf, 10.7 GB, sha25647b627b7…4c694cb, refused arms (IQ2_S/IQ2_XXS/Q3_K) named beside it.Gates (operator-rerun numbers)
test cases: 1 | 1 passed | 0 failed,assertions: 116 | 116 passed— bit-exact vsVecDotIQ3_XXSQ8_Kon the P150.1 | 1 passed,13 | 13 assertions— and it goes RED under the reviewer's dispatch-only mutation (kIq3_xxs has no GROUPED keep-quant decode).74 | 74 passed,assertions: 524455 | 524455; fulltest_gguf_keep_quant61 | 61,12129 | 12129.What is NOT reached (owed, tracked)
The APEX-I-Nano e2e generation gate is recorded OOM/owed: the run fatals on the PRE-EXISTING Q4_K grouped decode's
kq-decode/repairplane (1,116,733,440 B ask; fatal 960 MB alloc, 108 MB largest free block) — APEX's 133 refused-arm Q3_K/IQ2_* tensors expand to bf16 and fragment the allocator. The IQ3_XXS staging itself is NOT the trigger (EnsureKeepQuantWordsdeltas 15–47 MB). Evidence: the alloc trace in the branch records; fix owner is the keep-quant row's W4 residency redesign, not this row. Spec## Owedalso carries the W4a grouped arm for IQ3_XXS and the IQ2_S/IQ2_XXS/Q3_K waves. Row state movesINVENTORIED→PARTIALwith code+test anchors.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]