Skip to content

Fix stack overflow when decoding deeply nested structures like arrays and maps#51

Merged
MarshalX merged 10 commits into
mainfrom
fix-overflow-on-deep-structures
Feb 17, 2025
Merged

Fix stack overflow when decoding deeply nested structures like arrays and maps#51
MarshalX merged 10 commits into
mainfrom
fix-overflow-on-deep-structures

Conversation

@MarshalX

@MarshalX MarshalX commented Nov 18, 2024

Copy link
Copy Markdown
Owner

fixes #9

It uses 2 approaches. First for Python 3.8 and second for Python 3.9+. The first one hard limits recursion to 1k (same as Python by default). The second one respects sys.setrecursionlimit. decode_dag_cbor function could now raise RecursionError

CI fails because of temporarily disabled endpoint by bsky team to deal with network load

use of Py_EnterRecursiveCall and Py_LeaveRecursiveCall is slow; so just count depth and check it using Py_GetRecursionLimit; raise PyRecursionError

@codspeed-hq

codspeed-hq Bot commented Nov 23, 2024

Copy link
Copy Markdown

CodSpeed Performance Report

Merging #51 will degrade performances by 28.99%

Comparing fix-overflow-on-deep-structures (1083c37) with main (7bf9fb9)

Summary

⚡ 4 improvements
❌ 8 regressions
✅ 180 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
test_dag_cbor_decode[roundtrip01.json] 23.2 µs 16.7 µs +39.2%
test_dag_cbor_decode[roundtrip22.json] 16.8 µs 22.7 µs -26%
test_dag_cbor_encode[roundtrip05.json] 22.9 µs 16.9 µs +35.38%
test_dag_cbor_encode[roundtrip06.json] 15.1 µs 21.3 µs -28.99%
test_dag_cbor_encode[roundtrip11.json] 22.9 µs 17.6 µs +29.73%
test_dag_cbor_encode[roundtrip12.json] 16.3 µs 18.2 µs -10.83%
test_dag_cbor_encode[roundtrip16.json] 16.9 µs 22.1 µs -23.69%
test_dag_cbor_encode[roundtrip20.json] 22.6 µs 16.6 µs +35.8%
test_dag_cbor_encode[roundtrip21.json] 16.1 µs 18.2 µs -11.26%
test_dag_cbor_encode[roundtrip25.json] 16 µs 22 µs -27.44%
test_dag_cbor_encode[roundtrip26.json] 16.1 µs 18.3 µs -12.15%
test_dag_cbor_encode_real_data[canada.json] 13.6 ms 15.4 ms -11.95%

@MarshalX

Copy link
Copy Markdown
Owner Author

R.I.P

image

@MarshalX

Copy link
Copy Markdown
Owner Author

okay, now it should not affect perf

@MarshalX MarshalX merged commit 744bc2f into main Feb 17, 2025
@MarshalX MarshalX deleted the fix-overflow-on-deep-structures branch February 17, 2025 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Segfault on decoding of nested lists and maps

1 participant