Pre-submission Checklist
GPU Hardware
2x Intel Arc Pro B70 (32 GB), bmg-g31-a0, Xe2 — 0000:09:00.0 and 0000:0d:00.0
DRI Devices Information
$ ls -ls /dev/dri/*
0 crw-rw---- 1 root video 226, 0 Aug 26 08:59 /dev/dri/card0
0 crw-rw---- 1 root video 226, 1 Aug 26 08:59 /dev/dri/card1
0 crw-rw---- 1 root video 226, 2 Aug 26 08:59 /dev/dri/card2
0 crw-rw---- 1 root video 226, 3 Aug 26 08:59 /dev/dri/card3
0 crw-rw---- 1 root render 226, 128 Aug 26 08:59 /dev/dri/renderD128
0 crw-rw---- 1 root render 226, 129 Aug 26 08:59 /dev/dri/renderD129
0 crw-rw---- 1 root render 226, 130 Aug 26 08:59 /dev/dri/renderD130
$ ls -la /dev/dri/by-path/
pci-0000:01:00.0-card -> ../card2 (not Intel)
pci-0000:01:00.0-render -> ../renderD129 (not Intel)
pci-0000:09:00.0-card -> ../card1 <- Arc Pro B70 #0
pci-0000:09:00.0-render -> ../renderD128
pci-0000:0d:00.0-card -> ../card3 <- Arc Pro B70 #1
pci-0000:0d:00.0-render -> ../renderD130
Note for anyone reading gtt_mm: each device appears three times under
/sys/kernel/debug/dri/ — PCI address, primary minor, render minor — so
globbing and summing triples the total. The reproducer matches only the
PCI-address form.
GPU Detailed Information (lspci output)
09:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics] (prog-if 00 [VGA controller])
Subsystem: Sparkle Computer Co., Ltd. Device 0105
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 95
IOMMU group: 20
Region 0: Memory at 3e00000000 (64-bit, prefetchable) [size=16M]
Region 2: Memory at 2800000000 (64-bit, prefetchable) [size=32G]
Expansion ROM at fb600000 [disabled] [size=2M]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Address: 00000000fee00000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [110 v1] Null
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable+, Smallest Translation Unit: 00
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
IOVSta: Migration-
Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: e223
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000003e01000000 (64-bit, prefetchable)
Region 2: Memory at 0000003000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [400 v1] Latency Tolerance Reporting
Max snoop latency: 1048576ns
Max no snoop latency: 1048576ns
Kernel driver in use: xe
Kernel modules: xe
Please disregard the 2.5GT/s x1 in the link registers. Both endpoints and
both downstream ports report it, in lspci and in sysfs
current_link_speed/max_link_speed, with runtime_status: active — yet
measured host-to-device bandwidth is 14.41 GB/s. The cards sit behind PCIe
switches and the real links are the switch uplinks, which the kernel reports
correctly at boot:
pci 0000:07:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s
PCIe x8 link at 0000:00:03.1 -> feeds 0000:09:00.0
pci 0000:0b:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s
PCIe x8 link at 0000:00:03.2 -> feeds 0000:0d:00.0
Measured bandwidth is 91% of theoretical on both, so the endpoint values are
simply wrong. Unrelated to this report either way — the mirror reproduces
identically on both cards despite their uplinks differing 2:1.
Key lines, both cards identical apart from addresses, IRQ and IOMMU group:
09:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Intel Graphics]
Subsystem: Sparkle Computer Co., Ltd. Device 0105
Region 2: Memory at 2800000000 (64-bit, prefetchable) [size=32G]
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1
LnkCtl: ASPM Disabled
LnkSta: Speed 2.5GT/s, Width x1
Capabilities: [420 v1] Physical Resizable BAR
BAR 2: current size: 32GB, supported: 256MB ... 32GB
Capabilities: [220 v1] Virtual Resizable BAR
BAR 2: current size: 8GB
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCtl: Enable- ... Number of VFs: 0
Kernel driver in use: xe
Kernel modules: xe
0d:00.0 — identical; Region 2 at 4000000000, IRQ 98, IOMMU group 25
Worth stating explicitly, as it pre-empts two obvious theories:
- ReBAR is fully enabled —
Physical Resizable BAR: BAR 2 current size: 32GB,
so all of VRAM is CPU-visible and the host mirror is not a small-BAR
workaround.
- SR-IOV is off (
xe.max_vfs=0, Number of VFs: 0) and ASPM is disabled.
The Virtual Resizable BAR ... 8GB entry is the inert VF BAR template.
Driver Version
26.27.39122.11-0 (libze-intel-gpu1, inside the container — this is the userspace that performs the allocations)
Installed GPU Driver Packages
--- container (Docker, performs the allocations; mounts no host libraries) ---
intel-igc-core-2 2.38.2
intel-igc-opencl-2 2.38.2
intel-ocloc 26.27.39122.11-0
libigdgmm12 22.10.0
libze-dev 1.32.0
libze-intel-gpu1 26.27.39122.11-0
libze1 1.32.0
/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1 -> libze_intel_gpu.so.1.15.39122
--- host (Proxmox VE / Debian 13; kernel driver only, userspace unused here) ---
intel-igc-core-2 2.40.13
libigdgmm12 22.10.0
libze-intel-gpu1 26.31.39395.13-0
libze1 1.32.0
Driver Installation Details
- Host (Proxmox VE): kernel-side `xe` driver in-tree; userspace installed from
the distribution repository on 2026-08-25:
apt install intel-igc-core-2 libigdgmm12 libigsc1 libmetee libze1 \
libze-intel-gpu1 xpu-smi
- Workload: Docker container inside an unprivileged LXC, /dev/dri passed
through. The container ships its own Level Zero and IGC and mounts no host
libraries. It is built from intel/vllm-xpu base images with oneAPI 2026.1.
- No custom kernel parameters for the GPU.
Linux Distribution
Other (please specify below)
Other Linux Distribution
Proxmox VE on Debian GNU/Linux 13 (trixie) Workload runs in a Docker container inside an unprivileged LXC on that host.
Kernel Version & Boot Parameters
$ uname -r
7.0.14-8-pve
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.14-8-pve root=/dev/mapper/pve-root ro quiet \
iommu=pt xe.max_vfs=0
$ lsmod | grep -iE "xe|i915|drm"
xe 4018176 58
drm_gpusvm_helper 57344 1 xe
intel_vsec 24576 1 xe
gpu_sched 69632 1 xe
drm_gpuvm 53248 1 xe
drm_buddy 28672 1 xe
drm_exec 12288 2 drm_gpuvm,xe
drm_suballoc_helper 16384 1 xe
drm_display_helper 286720 1 xe
drm_ttm_helper 20480 2 nvidia_drm,xe
ttm 126976 2 drm_ttm_helper,xe
SR-IOV is disabled (xe.max_vfs=0). An unrelated NVIDIA GPU is present on the
host for a separate VM; it shares only drm_ttm_helper.
Actual Behavior
When two Intel discrete GPUs share a SYCL context, every device allocation
receives a host-memory mirror of equal size, mapped through the kernel driver's
GTT. The mirror is charged to the kernel, so it does not appear in `free`, `ps`,
`VmRSS`, or cgroup accounting — only in /sys/kernel/debug/dri/*/gtt_mm and
/proc/<pid>/fdinfo/*/drm-total-gtt.
Allocating 8 GiB on each of two cards from one process:
xpu:0 +8 GiB 0000:09:00.0 gtt=8.1 0000:0d:00.0 gtt=0.1
xpu:1 +8 GiB 0000:09:00.0 gtt=8.1 0000:0d:00.0 gtt=8.1
16 GiB allocated, 16.2 GiB of GTT. MemAvailable falls by 17.5 GiB and recovers
fully on free, so these are physical pages rather than aperture reservations.
VRAM is consumed normally as well (8.2 GiB per card via xpu-smi), so it is a
duplicate backing store, not device memory placed in system RAM.
The trigger is the shared context, not the presence of two GPUs:
one process, both devices 16 GiB allocated -> 16.2 GiB GTT
one process, one device 8 GiB allocated -> 0.1 GiB GTT
two processes, one each 16 GiB allocated -> 0.2 GiB GTT
The third row is the same two cards, the same total allocation, at the same
moment. Only the context topology differs.
In practice this means a 32 GiB card requires 32 GiB of system memory alongside
it. On this two-card host it consumed ~58 GiB and drove a 125 GiB machine into
swap with 13 of 15 GiB used.
Expected Behavior
Device allocations in a multi-device context should not require a host-memory
backing store of equal size, or there should be a documented way to opt out for
allocations that will never be accessed from a peer device.
Per-process device isolation already achieves this — 0.2 GiB rather than
16.2 GiB for the same work — so the mirror does not appear to be required by
the hardware.
Reproduction Rate
Always reproduces - 100%
Steps to Reproduce
1. Two Intel discrete GPUs visible to one process.
2. Run the script below (PyTorch XPU; the same behaviour was reported via
sycl::malloc_device in ggml-org/llama.cpp#22116).
3. Watch /sys/kernel/debug/dri/<pci>/gtt_mm, or /proc/self/fdinfo/*/drm-total-gtt
from inside the process.
4. GTT rises 1:1 with device allocation on the allocating card, and MemAvailable
falls with it.
5. Re-run with ONEAPI_DEVICE_SELECTOR=level_zero:0 — GTT stays at ~0.1 GiB.
6. Re-run as two processes, one device each — GTT stays at ~0.1 GiB per card.
See Source Code / Reproducer below for the script used in step 2.
Is this a regression?
Last Known Working Driver Version
No response
First Known Failing Driver Version
No response
API Call Logs
No response
strace Logs
No response
System Logs / dmesg Output
No response
Backtrace (if crash or hang occurred)
No response
Source Code / Reproducer
Standalone. No collectives, no distributed init, no multi-process machinery —
plain PyTorch XPU allocation.
import glob, re, time, torch
GiB = 1 << 30
def gtt():
out = {}
for p in sorted(glob.glob("/sys/kernel/debug/dri/*/gtt_mm")):
# each device appears three times in debugfs: PCI address,
# primary minor, render minor. Match only the PCI-address form.
if not re.match(r"^[0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9]$", p.split("/")[-2]):
continue
for line in open(p):
if "usage:" in line:
out[p.split("/")[-2]] = round(int(line.split()[1]) / GiB, 1)
return out
held = []
for dev in range(torch.xpu.device_count()):
for _ in range(4):
held.append(torch.empty(2 * GiB, dtype=torch.uint8, device=f"xpu:{dev}"))
torch.xpu.synchronize(); time.sleep(0.5)
print(f"xpu:{dev}", gtt(), flush=True)/proc/self/fdinfo/*/drm-total-gtt reports the same per process and needs no
debugfs, which is how llama.cpp#22116 originally found it — the mapping is
invisible to VmRSS.
Command Line / Application Details
python3 gtt-repro.py # both devices visible
ONEAPI_DEVICE_SELECTOR=level_zero:0 python3 gtt-repro.py # single-device control
# two processes, one device each — the case that shows the trigger is the
# shared context rather than the number of GPUs present
for i in 0 1; do ONEAPI_DEVICE_SELECTOR=level_zero:$i python3 gtt-repro.py & done; wait
Originally found under vLLM at tensor-parallel-size 2, where it cost ~58 GiB of
host RAM. The reproducer above is the minimal form.
oneAPI Version (if applicable)
oneAPI 2026.1 (container). PyTorch XPU.
Level Zero loader libze1 1.32.0; runtime libze-intel-gpu1 26.27.39122.11-0.
Screenshots / Video
No response
Additional Notes
The single-device and two-process controls are the useful part of this report.
They narrow the trigger from "two GPUs are present" to "one context spans two
devices", which is a much smaller thing to reason about, and they show the
behaviour is avoidable without any change to how much is allocated.
Question: is the host mirror required for multi-device contexts on hardware
without peer-to-peer, or is it an unintended default? If it is required, is
there a way to opt out for allocations that will never be accessed from a peer
device?
For context on why the second form matters: giving each worker process its own
single-device context took a two-card vLLM server from 68.2 GiB of GTT to
21.5 GiB, with the remainder being a host-memory buffer that is legitimately
mapped into both cards. Prior report on the same hardware via
sycl::malloc_device, closed stale:
ggml-org/llama.cpp#22116.
Pre-submission Checklist
GPU Hardware
2x Intel Arc Pro B70 (32 GB), bmg-g31-a0, Xe2 — 0000:09:00.0 and 0000:0d:00.0
DRI Devices Information
Note for anyone reading
gtt_mm: each device appears three times under/sys/kernel/debug/dri/— PCI address, primary minor, render minor — soglobbing and summing triples the total. The reproducer matches only the
PCI-address form.
GPU Detailed Information (lspci output)
Key lines, both cards identical apart from addresses, IRQ and IOMMU group:
Worth stating explicitly, as it pre-empts two obvious theories:
Physical Resizable BAR: BAR 2 current size: 32GB,so all of VRAM is CPU-visible and the host mirror is not a small-BAR
workaround.
xe.max_vfs=0,Number of VFs: 0) and ASPM is disabled.The
Virtual Resizable BAR ... 8GBentry is the inert VF BAR template.Driver Version
26.27.39122.11-0 (libze-intel-gpu1, inside the container — this is the userspace that performs the allocations)
Installed GPU Driver Packages
Driver Installation Details
Linux Distribution
Other (please specify below)
Other Linux Distribution
Proxmox VE on Debian GNU/Linux 13 (trixie) Workload runs in a Docker container inside an unprivileged LXC on that host.
Kernel Version & Boot Parameters
SR-IOV is disabled (
xe.max_vfs=0). An unrelated NVIDIA GPU is present on thehost for a separate VM; it shares only
drm_ttm_helper.Actual Behavior
Expected Behavior
Reproduction Rate
Always reproduces - 100%
Steps to Reproduce
See Source Code / Reproducer below for the script used in step 2.
Is this a regression?
Last Known Working Driver Version
No response
First Known Failing Driver Version
No response
API Call Logs
No response
strace Logs
No response
System Logs / dmesg Output
No response
Backtrace (if crash or hang occurred)
No response
Source Code / Reproducer
Standalone. No collectives, no distributed init, no multi-process machinery —
plain PyTorch XPU allocation.
/proc/self/fdinfo/*/drm-total-gttreports the same per process and needs nodebugfs, which is how llama.cpp#22116 originally found it — the mapping is
invisible to
VmRSS.Command Line / Application Details
Originally found under vLLM at tensor-parallel-size 2, where it cost ~58 GiB of
host RAM. The reproducer above is the minimal form.
oneAPI Version (if applicable)
Screenshots / Video
No response
Additional Notes
The single-device and two-process controls are the useful part of this report.
They narrow the trigger from "two GPUs are present" to "one context spans two
devices", which is a much smaller thing to reason about, and they show the
behaviour is avoidable without any change to how much is allocated.
Question: is the host mirror required for multi-device contexts on hardware
without peer-to-peer, or is it an unintended default? If it is required, is
there a way to opt out for allocations that will never be accessed from a peer
device?
For context on why the second form matters: giving each worker process its own
single-device context took a two-card vLLM server from 68.2 GiB of GTT to
21.5 GiB, with the remainder being a host-memory buffer that is legitimately
mapped into both cards. Prior report on the same hardware via
sycl::malloc_device, closed stale:ggml-org/llama.cpp#22116.