Skip to content

Add NUMA balancing detection and performance warning#679

Open
Functionhx wants to merge 1 commit into
deepseek-ai:mainfrom
Functionhx:fix/numa-balancing-warning
Open

Add NUMA balancing detection and performance warning#679
Functionhx wants to merge 1 commit into
deepseek-ai:mainfrom
Functionhx:fix/numa-balancing-warning

Conversation

@Functionhx

@Functionhx Functionhx commented Jul 9, 2026

Copy link
Copy Markdown

@ds-review-bot

Copy link
Copy Markdown
Collaborator

🤖 ds-review-bot Code Review

v4

⚠️ 未完成评审(no_result_file:模型未产出结果文件)

v3

This is a well-scoped, low-risk PR that adds NUMA balancing detection with a non-fatal performance warning at import time in deep_ep/init.py. The implementation is correct and idiomatic. Strengths: (1) It mirrors the existing EP_SUPPRESS_NCCL_CHECK pattern including the int(os.environ.get(..., 0)) idiom. (2) It correctly uses warnings.warn() rather than assert, matching the intent that NUMA balancing is a performance concern (non-fatal), unlike check_nccl_so() which correctly aborts; good use of stacklevel=2 and an actionable remediation command in the message. (3) except OSError is the right choice: it covers both FileNotFoundError (non-Linux / no procfs) and PermissionError (containers where the file exists but is unreadable), since both subclass OSError, and the != '0' comparison correctly treats tiered modes (1, 2) as enabled. (4) Placement before check_nccl_so() and init_jit() ensures the warning surfaces early at import time. The main gap is documentation: the new EP_SUPPRESS_NUMA_CHECK environment variable is not added to the README, unlike the existing EP_SUPPRESS_NCCL_CHECK which is documented. Recommend adding that one line before merge; otherwise approve.

Files reviewed: 1
Issues found: 🔵 1 suggestion
Inline comments posted: 0
General comments (无法定位到 diff): 1

⚠️ Parse warning: [v4] no_result_file:模型未产出结果文件


📍 未定位到 diff 的评论

🔵 suggestion README.md:L337: The new EP_SUPPRESS_NUMA_CHECK env var is not documented. The sibling EP_SUPPRESS_NCCL_CHECK is documented here under 'Environment variables > General'. For consistency, add a line such as: '- EP_SUPPRESS_NUMA_CHECK: 0 or 1, suppress NUMA balancing detection warning, 0 by default'. This PR only touched deep_ep/init.py. 🤖 v3

NUMA balancing auto-migrates pages between NUMA nodes, which causes severe
performance degradation for bandwidth-intensive internode dispatch/combine
kernels that use RDMA and NVLink. Detect when /proc/sys/kernel/numa_balancing
is non-zero at import time and emit a clear warning telling the user how to
disable it.

Test Plan:
- Verified that `import deep_ep` emits a warning when
  /proc/sys/kernel/numa_balancing contains "1"
- Verified that `import deep_ep` is silent when
  /proc/sys/kernel/numa_balancing contains "0" or is absent

Signed-off-by: Yuchen Fan <functionhx@gmail.com>
@Functionhx
Functionhx force-pushed the fix/numa-balancing-warning branch from d37d89c to b55e3c1 Compare July 10, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants