Skip to content

fix(proxy): raise ValueError for unknown IP_PROXY_PROVIDER_NAME instead of crashing with AttributeError - #925

Merged
NanmiCoder merged 1 commit into
NanmiCoder:mainfrom
kapil971390:fix/proxy-provider-unknown-name-error
Jul 1, 2026
Merged

fix(proxy): raise ValueError for unknown IP_PROXY_PROVIDER_NAME instead of crashing with AttributeError#925
NanmiCoder merged 1 commit into
NanmiCoder:mainfrom
kapil971390:fix/proxy-provider-unknown-name-error

Conversation

@kapil971390

Copy link
Copy Markdown
Contributor

问题

config.pyIP_PROXY_PROVIDER_NAME 被设置为一个不存在于 IpProxyProvider 字典中的值时(例如把 "kuaidaili" 拼写成 "kuaidalli"),IpProxyProvider.get() 会返回 None,这个 None 被静默地传入 ProxyIpPool.ip_provider

程序在启动时没有任何报错,直到调用 load_proxies() 时才崩溃:

AttributeError: 'NoneType' object has no attribute 'get_proxy'

此错误信息完全不提示根本原因,用户很难判断问题出在 IP_PROXY_PROVIDER_NAME 配置上。

复现步骤

  1. config/base_config.py 中将 IP_PROXY_PROVIDER_NAME 改为错误值,例如 "kuaidalli"
  2. ENABLE_IP_PROXY = True
  3. 启动任意爬虫
  4. create_ip_pool()load_proxies() 抛出 AttributeError: 'NoneType' object has no attribute 'get_proxy'

方案

create_ip_pool() 中,.get() 取到 None 时立即抛出 ValueError,并在错误信息中列出有效选项:

# proxy/proxy_ip_pool.py — before
ip_provider=IpProxyProvider.get(config.IP_PROXY_PROVIDER_NAME),  # None if unknown

# after
ip_provider = IpProxyProvider.get(config.IP_PROXY_PROVIDER_NAME)
if ip_provider is None:
    raise ValueError(
        f"Unknown proxy provider: '{config.IP_PROXY_PROVIDER_NAME}'. "
        f"Valid options: {list(IpProxyProvider.keys())}"
    )

修复后:

ValueError: Unknown proxy provider: 'kuaidalli'. Valid options: ['kuaidaili', 'wandouhttp', 'static']

测试

test/test_proxy_ip_pool.py 新增回归测试,覆盖拼写错误、空字符串、大小写错误及三个合法 provider 名称不受影响。


🤖 Generated with Claude Code

When IP_PROXY_PROVIDER_NAME in config.py is set to an unrecognised
value (e.g. typo "kuaidalli" instead of "kuaidaili"), IpProxyProvider.get()
returns None which is silently stored in ProxyIpPool.ip_provider.  The
crash surfaces later as:

  AttributeError: 'NoneType' object has no attribute 'get_proxy'

with no hint about what went wrong or how to fix it.

Add an explicit None-check in create_ip_pool() that raises ValueError
with the bad name and the list of valid options, so the root cause is
immediately visible.  Also add a regression test covering the four
invalid-name edge cases (typo, empty string, wrong case, unknown name)
and the three valid names to confirm they are unaffected.
@kapil971390
kapil971390 requested a review from NanmiCoder as a code owner June 26, 2026 06:44
@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. bug Something isn't working labels Jun 26, 2026
@kapil971390

kapil971390 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Reproduction screenshots

Before fixip_provider=None stored silently, crash surfaces only on load_proxies():

mediacrawler-proxy-BEFORE

After fixValueError raised immediately with actionable message:

mediacrawler-proxy-AFTER

@NanmiCoder

Copy link
Copy Markdown
Owner

Thanks

@NanmiCoder
NanmiCoder merged commit 98a2830 into NanmiCoder:main Jul 1, 2026
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working lgtm This PR has been approved by a maintainer size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants