Skip to content

URL Length check for locate_data#3356

Merged
bsipocz merged 4 commits into
astropy:mainfrom
d-giles:3276-longURI
Jul 21, 2025
Merged

URL Length check for locate_data#3356
bsipocz merged 4 commits into
astropy:mainfrom
d-giles:3276-longURI

Conversation

@d-giles

@d-giles d-giles commented Jun 24, 2025

Copy link
Copy Markdown
Contributor

Fixes #3276.

Considering a standard URL character limit of ~2000 for GET requests, this PR checks the number of targets being queried. Assuming a maximum ID length of 16 (based on the heuristic of not having seen IDs approaching this length personally...), the query could support ~100 IDs at a time before running into the character limit for the URL. This PR splits a query into multiple chunks to avoid this limit and combines the result into a single table.

This PR is created as a draft as the assumptions I've made could merit further discussion, and I'm also unsure if this is the desired behavior rather than simply failing when running into a query that is too long.

@codecov

codecov Bot commented Jun 24, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 70.07%. Comparing base (82cab62) to head (51ef0b8).
⚠️ Report is 145 commits behind head on main.

Files with missing lines Patch % Lines
astroquery/heasarc/core.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3356   +/-   ##
=======================================
  Coverage   70.07%   70.07%           
=======================================
  Files         232      232           
  Lines       19893    19893           
=======================================
  Hits        13940    13940           
  Misses       5953     5953           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@keflavich

Copy link
Copy Markdown
Contributor

This looks good to me, and I like the idea of breaking up queries that are too long. As long as HEASARC is OK with getting a bunch of split up GET requests, this seems like a good solution.

Just to check - there isn't a different API that accepts large POST requests, right? That would be a simpler fallback if so.

@d-giles

d-giles commented Jun 25, 2025

Copy link
Copy Markdown
Contributor Author

Just to check - there isn't a different API that accepts large POST requests, right? That would be a simpler fallback if so.

In fact there is, I've updated the PR to bypass the execute method of a dal query to use the execute_votable method using post=True.

@bsipocz bsipocz added this to the v0.4.11 milestone Jun 25, 2025
@bsipocz

bsipocz commented Jun 25, 2025

Copy link
Copy Markdown
Member

Sounds all this good, flip the PR status out from draft when you think this is ready for in depth review/merge.

@d-giles
d-giles marked this pull request as ready for review June 25, 2025 19:01

@bsipocz bsipocz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bsipocz
bsipocz merged commit 9073293 into astropy:main Jul 21, 2025
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: Heasarch.locate_data() triggers 414 Client Error: Request-URI Too Large

3 participants