Skip to content

[codex] Fix data parser cache leaks with attribution#736

Merged
ObservedObserver merged 3 commits into
mainfrom
codex/fix-data-parser-cache-leak-attributed
Jun 22, 2026
Merged

[codex] Fix data parser cache leaks with attribution#736
ObservedObserver merged 3 commits into
mainfrom
codex/fix-data-parser-cache-leak-attributed

Conversation

@ObservedObserver

Copy link
Copy Markdown
Member

Summary

  • Preserve the original Fix memory leak by replacing @lru_cache with instance-level caching in data parsers #724 commits from Lilo so the initial fix authorship is retained.
  • Replace method-level @lru_cache() on data parser cached properties with per-instance cached_property in the final implementation.
  • Add regression coverage proving pandas parser cached properties do not retain parser/DataFrame instances and that parsers remain picklable after cache loading.

Root Cause

functools.lru_cache stores calls in the decorated function wrapper. For instance methods and properties, that cache key includes self, so parser instances are retained by the class-level cache after callers drop their references. Those parser instances keep their DataFrames alive, causing memory growth in long-running processes that parse many datasets.

Notes

This PR carries forward the original work from #724, then adds a compatibility adjustment: the final implementation uses cached_property instead of storing a threading.Lock on parser instances, so parser pickling keeps working.

Supersedes #735.
Fixes #723.

Validation

  • ./venv/bin/python -m pytest tests/test_data_parsers.py -q
  • ./venv/bin/python -m pytest -q
  • ./venv/bin/python -m ruff check .

@ObservedObserver
ObservedObserver merged commit d1ec1e0 into main Jun 22, 2026
14 checks passed
@ObservedObserver
ObservedObserver deleted the codex/fix-data-parser-cache-leak-attributed branch June 22, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Memory Leak in Data Parsers: Instance Methods Decorated with @lru_cache Retain Large DataFrames

2 participants