Complete the backend-neutral HTTP protocol and migrate intgs onto it by mwdd146980 · Pull Request #24516 · DataDog/integrations-core

mwdd146980 · 2026-07-10T21:16:53Z

What does this PR do?

Extends the backend-neutral HTTP surface with the client and response members integrations reach for today, and implements them on RequestsWrapper.
Drops the Protocol suffix from the two classes (HTTPClientProtocol -> HTTPClient, HTTPResponseProtocol -> HTTPResponse).
Adds an overridable AgentCheck.create_http_client() factory behind self.http, so the HTTP backend can be swapped without touching consumers. The default still returns a RequestsWrapper.
Moves MockHTTPResponse out of the shipped base wheel into datadog_checks_dev, enforces the HTTPResponse surface on it, and updates its import sites.
Adds a backend-neutral exception surface and migrates every in-tree consumer that caught requests-specific exceptions onto it.
Migrates the last consumers that reached into raw requests objects:
- The kubelet base check parses its pod list through response.json().
- http_check reads the TLS peer certificate through get_peer_cert().
- openstack_controller's SDK path drops its direct requests coupling.

The requests backend is unchanged. Every protocol addition is additive, and the exception migration preserves the same catch behavior through the neutral aliases.

Motivation

Groundwork for the httpx backend. Checks must depend on a backend-neutral HTTP surface rather than on requests objects, so the backend can be swapped later without editing each integration. An audit of what integrations use today surfaced gaps in the client and response contracts and a long tail of checks catching requests.exceptions.* directly. This fills the contract gaps and moves those consumers onto the neutral surface.

Relocating MockHTTPResponse to datadog_checks_dev also stops shipping test-only code inside the Agent bundle.

Verification

New unit tests cover the added client and response capabilities on RequestsWrapper, the enforced MockHTTPResponse surface, the kubelet response.json() path, and the http_check peer-cert read.
The exception migration is covered by each integration's existing error-handling tests, updated to raise and assert the neutral exception types.
Changelog entries are deferred until master-merge prep, so none ship in this branch yet.

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit).
Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
Backport not required.

datadog-prod-us1-4 · 2026-07-10T21:19:29Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 77.49%
• Overall Coverage: 88.29% (-0.09%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5d79646 | Docs | Datadog PR Page | Give us feedback!}

…name Follow the base rename in PR #24516: VoltDBAuth now types against HTTPRequest (imported from datadog_checks.base.utils.http). Also merges the updated #24516 base so this stacked branch builds against the renamed, repaired foundation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f1cfae991e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…seam Freeze the backend-neutral HTTP surface in http_protocol.py and implement it on RequestsWrapper so consumers depend on the protocol, not requests: - Add connection close(), get_cookie(), and a trust_env property to HTTPClientProtocol/RequestsWrapper (capabilities expose behavior, never a backend object). - Add an agnostic auth hook (HTTPRequestAuth + OutgoingRequest) with a requests.auth.AuthBase adapter, so checks contribute headers/params without a requests-family auth type. - Add an overridable AgentCheck.create_http_wrapper() factory seam behind self.http, so the backend is swappable without touching consumers. Backend stays requests; changes are additive. Foundation for the httpx2 backend.

Address code-review findings on the agnostic HTTP surface: - get_cookie: return default without eagerly building a session when none exists yet, and catch requests CookieConflictError so the documented value-or-default contract holds even with duplicate cookie names; fix the docstring. - Annotate the new _OutgoingRequest and _RequestsAuthHookAdapter helpers, matching the typed capabilities and protocols added in the same change. - Add regression tests: idempotent close after open, no-eager-session on cookie lookup, and conflict-returns-default.

- Trim the _RequestsAuthHookAdapter docstring to describe actual behavior: header edits apply via the shared dict, params are merged into the URL query, and the URL is read-only (finding 1). - Document that only per-call auth= is adapted for HTTPRequestAuth hooks; a hook set persistently via options['auth'] is unsupported (finding 4). - Revert the get_cookie hardening back to the minimal RequestsCookieJar.get delegation; that robustness was out of scope for a behavior-preserving foundation PR (finding 3). Keeps the new-code type annotations (finding 5) and the idempotent-close test (finding 7).

…otocol Round-2 review follow-ups on the agnostic HTTP surface: - get_cookie: catch requests CookieConflictError and return default so the method honors its own documented value-or-default contract. The method is new and the frozen protocol declares that contract, and the exception is requests-specific with no httpx2 equivalent, so leaving it unhandled would diverge per backend. Adds a cross-domain collision regression test. - OutgoingRequest protocol: document that url is inspection-only and writes are not guaranteed to be honored, in the frozen contract file where a second backend author will read it.

When a session is injected via session=, initialize the tracked _trust_env from that session's trust_env instead of hard-coding True, so the reported trust_env matches the session it wraps. getattr guards sessions that do not expose the attribute (e.g. autospec mocks). Adds a regression test for the injected-session case. Addresses round-2 review finding 2.

…_wrapper to create_http_client Rename the agnostic auth-hook request view OutgoingRequest (and its concrete _OutgoingRequest) to HTTPRequest/_HTTPRequest, and the factory seam AgentCheck.create_http_wrapper to create_http_client, aligning with the HTTPClientProtocol return type. Behavior unchanged.

ruff isort reordered the renamed http_protocol imports (HTTPRequest before HTTPRequestAuth) in http.py and test_auth_hook.py, and the double-close test's bare 'http.session' expression is now an assertion to satisfy B018.

The HTTPClientProtocol.get_cookie contract said only value or default; the concrete RequestsWrapper returns default on an ambiguous same-name cookie (CookieConflictError) but the frozen backend-neutral protocol gave no signal that a second backend must do the same. Align the contract so a future backend implements the edge case identically.

Remove HTTPRequest/HTTPRequestAuth from the protocol surface and the _HTTPRequest/_RequestsAuthHookAdapter plumbing plus its per-call dispatch, and delete the seam-only test. The seam had a single intended consumer (voltdb), which now sends credentials as query params without it, so the frozen surface should not carry the hook. Net diff of this PR no longer introduces it.

… into mwdd146980/http-agnostic-protocol-surface

MockHTTPResponse's __getattr__/__setattr__ rejected any name not on the HTTPResponse protocol, including leading-underscore names. When wrapped by wrapt's ResponseWrapper, wrapt forwards private attribute sets (e.g. _ResponseWrapper__default_chunk_size) to the wrapped object, and the guard raised AttributeError on those, breaking the request path. A protocol is a public contract, so only public names should be checked against it; underscore-prefixed names pass through.

Swaps the .raw.connection.sock.getpeercert access for the HTTPResponse.get_peer_cert protocol member, so the check no longer depends on the requests-specific raw urllib3 response.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Declare ignore_tls_warning and persist_connections as writable client attributes, add a should_bypass_proxy capability, and document the per-request persist option on the verb methods. These cover the client touchpoints (items 2-5 of the surface audit) integrations reach for today, so they can move onto the agnostic protocol without reaching into RequestsWrapper internals. RequestsWrapper already carried the two attributes as slots and consumed persist in _request; only should_bypass_proxy is new, delegating to the existing module-level helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Shorten multi-line comments to single lines and drop backticks.

Shorten the comment to one line and drop the semicolon.

Shorten multi-line comments to single lines and drop backticks.

Run ddev test --fmt across the PR's changed files.

Entries land on master through feature branch #22676, so the changelog filenames must reference that PR number. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Agnosticize integrations batch A1 (catch-narrowing + tests) Narrow self.http catch sites to backend-agnostic http_exceptions and swap interception-path tests to the agnostic surface for: activemq_xml, ambari, arangodb, citrix_hypervisor, cloud_foundry_api, consul. * Agnosticize integrations batch A2 (catch-narrowing + tests) Same agnosticization for: control_m, couch, couchbase, druid, ecs_fargate, elastic, envoy. * Preserve error handling after agnostic exception narrowing (batch A review) Address review findings on #24546 where narrowed catch sites changed behavior versus master: - couchbase: the agnostic ResponseWrapper.json() raises a stdlib json.JSONDecodeError (ValueError), not an HTTPError, so a 200 with a non-JSON body escaped `except HTTPError` and aborted the run. Master caught it via the broad RequestException. Also catch json.JSONDecodeError at the three sync-gateway/query-monitoring/index-stats sites. - activemq_xml: ConnectTimeout translates to HTTPTimeoutError (a sibling of HTTPConnectionError), so connect timeouts escaped `except HTTPConnectionError` and defeated suppress_errors. Catch HTTPTimeoutError as well. - druid: add explicit parametrize ids to the can-connect failure test. * chore: drop changelog entries for feature-branch merge This PR merges into a feature branch, not master, so changelog entries aren't needed until the feature branch itself ships. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Agnosticize integrations batch B (catch-narrowing + tests) Narrow self.http catch sites to backend-agnostic http_exceptions and swap interception-path tests to the agnostic surface for: fly_io, gitlab, harbor, hdfs_datanode, hdfs_namenode, ibm_db2, kubelet, kyototycoon, mapreduce, marathon, marklogic, mesos_master. * Catch connect timeouts in marklogic after agnostic narrowing (batch B review) ConnectTimeout translates to HTTPTimeoutError, a sibling of HTTPConnectionError, so it escaped `except (HTTPStatusError, HTTPConnectionError)` and skipped the CRITICAL connect service check that master emitted (master caught it via requests ConnectionError). Add HTTPTimeoutError to the catch. * Rename kubelet node-metrics test to match agnostic exception (batch 2 review) The test body raises HTTPStatusError, not a requests HTTPError, so rename test_report_node_metrics_kubernetes1_18_requests_httperror to ..._httpstatuserror. Test-only, no behavior change. * chore: drop changelog entries for feature-branch merge This PR merges into a feature branch, not master, so changelog entries aren't needed until the feature branch itself ships. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Agnosticize integrations batch C1 (catch-narrowing + tests) Narrow self.http catch sites to backend-agnostic http_exceptions and swap interception-path tests to the agnostic surface for: mesos_slave, n8n, nifi, nutanix, octopus_deploy. Keeps the existing n8n/nutanix changelog entries already present on the base. * Agnosticize integrations batch C2 (catch-narrowing + tests) Narrow self.http catch sites to backend-agnostic http_exceptions and swap interception-path tests to the agnostic surface for: proxmox, rabbitmq, sonarqube, sonatype_nexus, spark, squid, teamcity, traefik_mesh, yarn, and complete prefect. * Preserve malformed-JSON handling in sonarqube after narrowing (batch C review) The agnostic ResponseWrapper.json() raises a stdlib json.JSONDecodeError (ValueError), not an HTTPError, so a 200 with a non-JSON body escaped `except HTTPError` in check() and skipped the connect service check that master emitted (master caught it via the broad RequestException). Also catch json.JSONDecodeError. * Tidy nifi test helpers after agnostic migration (batch 3 review) Address non-blocking review nits on the new nifi test helpers: - Drop the leading underscore from the module-level helpers introduced in this PR (_dispatch, _mock_http_responses -> dispatch, mock_http_responses) per the repo naming convention for test-file helpers. Pre-existing underscore helpers are left untouched (out of scope). - Route test_overlapping_process_groups_no_duplicates through mock_http_responses instead of a hand-rolled side_effect, matching its siblings. Behavior is identical (same URL-substring dispatch). * chore: drop changelog entries for feature-branch merge This PR merges into a feature branch, not master, so changelog entries aren't needed until the feature branch itself ships. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…4594) * fix(openstack_controller): use backend-neutral HTTP protocol for keystone session Build the keystoneauth Session from the backend-neutral HTTPClient protocol surface (verify, cert, headers) instead of handing keystoneauth the underlying requests.Session. This removes the last openstack_controller coupling to the requests backend while preserving TLS config and the Nova and Ironic microversion headers on the SDK data-plane connection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(openstack_controller): add changelog entry for 24594 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Delete openstack_controller/changelog.d/24594.fixed --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Two stragglers from this branch's own HTTP refactors were left with stale references and broke CI: - MockHTTPResponse moved from datadog_checks.base.utils.http_testing to datadog_checks.dev.http; update the four test files still importing the old path (sonatype_nexus, nifi, traefik_mesh, nutanix). - HTTPResponseProtocol was renamed to HTTPResponse in http_protocol; update the two production files still referencing the old name (nutanix at runtime, control_m under TYPE_CHECKING). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Close protocol-compliance and hardening gaps surfaced by the automated self review, and finish decoupling http_check from requests internals. - Wrap ResponseWrapper.history so redirect items satisfy HTTPResponse and translate exceptions, instead of leaking raw requests objects. - Guard get_peer_cert against bare non-TLS sockets that lack getpeercert, so plain http:// requests return None rather than raise. - Route http_check through trust_env and close() instead of reaching into the raw session, removing its last requests-specific internals. - Drop the leading underscore from CaseInsensitiveDict and prune the redundant dunders from protocol_members(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

This PR merges into the httpx-migration-base feature branch, not master, so per-integration changelog entries are premature. Remove the entries added here and restore the base-branch entries this branch had deleted, leaving the PR changelog-neutral.

…24597) * fix(http): migrate remaining consumers to backend-neutral HTTP protocol Replace the last direct reaches into the requests Session across four integrations with the backend-neutral HTTPClient protocol: - avi_vantage: http.close() and http.get_cookie() - cisco_aci: http.close() - hpe_aruba_edgeconnect: http.get_cookie() and http.set_header() - http_check: http.trust_env setter and http.close() Behavior is preserved; only the access path changes. Updates the hpe_aruba_edgeconnect unit tests to drive the protocol methods. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

dd-octo-sts · 2026-07-17T16:53:33Z

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and code coverage settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

mwdd146980 added the qa/required QA is required for this PR and will generate a QA card label Jul 10, 2026

dd-octo-sts Bot added the base_package label Jul 10, 2026

mwdd146980 self-assigned this Jul 10, 2026

mwdd146980 added qa/skip-qa Automatically skip this PR for the next QA and removed qa/required QA is required for this PR and will generate a QA card labels Jul 10, 2026

mwdd146980 mentioned this pull request Jul 10, 2026

Migrate voltdb off the requests library for HTTP auth #24518

Open

3 tasks

Base automatically changed from mwdd146980/floor-exception-translation to mwdd146980/httpx-migration-base July 13, 2026 14:02

mwdd146980 mentioned this pull request Jul 13, 2026

httpx2 HTTP client backend (HTTPX2Wrapper) #24539

Closed

3 tasks

mwdd146980 changed the title ~~Base HTTP-agnostic protocol capabilities and auth/factory seam~~ Base HTTP-agnostic protocol capabilities and factory seam Jul 13, 2026

mwdd146980 marked this pull request as ready for review July 13, 2026 21:04

mwdd146980 requested a review from a team as a code owner July 13, 2026 21:04

dd-octo-sts Bot added the team/agent-integrations label Jul 13, 2026

chatgpt-codex-connector Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread datadog_checks_base/datadog_checks/base/utils/http.py

mwdd146980 added 10 commits July 13, 2026 17:55

style(base): fix ruff import-order and B018 after the rename

6bebc9a

ruff isort reordered the renamed http_protocol imports (HTTPRequest before HTTPRequestAuth) in http.py and test_auth_hook.py, and the double-close test's bare 'http.session' expression is now an assertion to satisfy B018.

docs(base): drop stale auth-hook re-export comment

cbb7c39

mwdd146980 force-pushed the mwdd146980/http-agnostic-protocol-surface branch from f1cfae9 to cbb7c39 Compare July 13, 2026 22:02

Merge remote-tracking branch 'origin/mwdd146980/httpx-migration-base'…

ed6c311

… into mwdd146980/http-agnostic-protocol-surface

This was referenced Jul 14, 2026

Agnosticize integrations HTTP error handling (batch 1 of 3) #24546

Merged

Agnosticize integrations HTTP error handling (batch 2 of 3) #24547

Merged

Agnosticize integrations HTTP error handling (batch 3 of 3) #24548

Merged

dd-octo-sts Bot added the integration/kubevirt_api label Jul 16, 2026

mwdd146980 and others added 14 commits July 16, 2026 11:49

feat(base): declare response attrs and get_peer_cert on protocol

a82d4d1

feat(base): add ResponseWrapper.get_peer_cert capability

62a26cf

feat(dev): enforce HTTPResponse surface on MockHTTPResponse

3fc77c0

fix(dev): add history attribute to mock response

7301cfc

fix(base): parse kubelet pod list via response.json instead of raw

41d3f2f

fix(http_check): read peer cert via get_peer_cert capability

6cdaa1c

Swaps the .raw.connection.sock.getpeercert access for the HTTPResponse.get_peer_cert protocol member, so the check no longer depends on the requests-specific raw urllib3 response.

chore: add changelog entries for HTTP response surface (#24516)

24674bd

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

chore(base): tighten HTTPResponse protocol comments

6f520ec

Shorten multi-line comments to single lines and drop backticks.

chore(dev): tighten MockHTTPResponse guard comment

3042004

Shorten the comment to one line and drop the semicolon.

chore(base): tighten HTTPClient protocol comments

63b5c05

Shorten multi-line comments to single lines and drop backticks.

style: fix import order flagged by ruff --fix

e02b626

Run ddev test --fmt across the PR's changed files.

chore: renumber changelog entries from 24516 to 22676

f384177

Entries land on master through feature branch #22676, so the changelog filenames must reference that PR number. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

triviajon approved these changes Jul 16, 2026

View reviewed changes

apiazza-dd approved these changes Jul 16, 2026

View reviewed changes

mwdd146980 and others added 3 commits July 17, 2026 09:21

NouemanKHAL mentioned this pull request Jul 17, 2026

Migrate openstack_controller to the backend-neutral HTTP protocol #24594

Merged

3 tasks

NouemanKHAL and others added 4 commits July 17, 2026 17:15

NouemanKHAL mentioned this pull request Jul 17, 2026

Migrate remaining integrations to the backend-neutral HTTP protocol #24597

Merged

3 tasks

This was referenced Jul 18, 2026

Decouple integration tests from the requests library #24602

Draft

Narrow HTTP exception catches in kube_* and openstack legacy #24604

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Complete the backend-neutral HTTP protocol and migrate intgs onto it#24516

Complete the backend-neutral HTTP protocol and migrate intgs onto it#24516
mwdd146980 wants to merge 36 commits into
mwdd146980/httpx-migration-basefrom
mwdd146980/http-agnostic-protocol-surface

mwdd146980 commented Jul 10, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-4 Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mwdd146980 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Verification

Review checklist (to be filled by reviewers)

Uh oh!

datadog-prod-us1-4 Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Validation Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mwdd146980 commented Jul 10, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Jul 10, 2026 •

edited

Loading