Skip to content

Use Arrow IO and memory API for better performance - #110

Merged
marcin-krystianc merged 4 commits into
masterfrom
dewv-20260506-perf
May 20, 2026
Merged

Use Arrow IO and memory API for better performance#110
marcin-krystianc merged 4 commits into
masterfrom
dewv-20260506-perf

Conversation

@marcin-krystianc

@marcin-krystianc marcin-krystianc commented May 8, 2026

Copy link
Copy Markdown
Contributor
  • Replace C++ standard library IO (fstream, ofstream, fopen/fread, ostringstream) and memory (std::vector<uint8_t>) with Arrow equivalents (arrow::io::ReadableFile, arrow::io::FileOutputStream, arrow::io::BufferOutputStream, arrow::ResizableBuffer)
  • Since arrow uses the mimalloc allocator, we can benefit from its arena allocator (memory reuse)
  • Eliminates platform-dependent pubsetbuf workarounds and reduces unnecessary buffer copies (e.g., ostringstream::str() + memcpy into vector<char> is now a single BufferOutputStream::Finish())
  • Update Cython bindings to pass arrow::Buffer directly instead of std::vector<char> + memoryview

@marcin-krystianc
marcin-krystianc marked this pull request as ready for review May 15, 2026 14:38
adamreeve
adamreeve previously approved these changes May 15, 2026

@adamreeve adamreeve left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good thanks Marcin, just noticed a minor typo

Comment thread python/palletjack/palletjack.cc Outdated
Co-authored-by: Adam Reeve <adam.reeve@gr-oss.io>
@marcin-krystianc
marcin-krystianc merged commit 47fb3b3 into master May 20, 2026
27 checks passed
@marcin-krystianc
marcin-krystianc deleted the dewv-20260506-perf branch May 20, 2026 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants