feat: BMI for bit filtering by devanbenz · Pull Request #10136 · apache/arrow-rs

devanbenz · 2026-06-12T13:09:48Z

This commit adds the ability for bit filtering to be done using the _pext_u64 BMI.

In this commit we check to see if the bitmap is dense or sparse prior to performing the pext operation, if we have a dense bitmap we just fallback to the original way of filtering.

We cannot run the BMI2 feature on the benchmark host machine -- here is the benchmarks from my machine:

Specs:

Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
Vendor ID:                   GenuineIntel
  Model name:                12th Gen Intel(R) Core(TM) i7-12700K

Benchmark	without bmi2	with bmi2	change
slices, kept 1023/1024	1.81 µs	1.74 µs	−4.1%
slices, kept 1023/1024, optimized	1.16 µs	0.98 µs	−15.9%
slices, kept 1023/1024, sliced	1.77 µs	1.68 µs	−5.4%
slices, kept 9/10	97.3 µs	3.28 µs	−96.6%
slices, kept 9/10, optimized	76.8 µs	3.29 µs	−95.7%
slices, kept 9/10, sliced	97.8 µs	3.41 µs	−96.5%
indices, kept 1/2	38.4 µs	2.26 µs	−94.1%
indices, kept 1/2, optimized	28.9 µs	2.29 µs	−92.0%
indices, kept 1/2, sliced	38.0 µs	2.28 µs	−94.0%
indices, kept 1/10	7.31 µs	2.56 µs	−65.0%
indices, kept 1/10, optimized	5.86 µs	2.54 µs	−56.7%
indices, kept 1/10, sliced	7.32 µs	2.07 µs	−71.8%
indices, kept 1/1024	0.69 µs	0.88 µs	+25.9% (regressed)
indices, kept 1/1024, optimized	95.1 ns	94.1 ns	within noise
indices, kept 1/1024, sliced	0.69 µs	0.85 µs	+23.1% (regressed)

…nges#diff-f2fe5b601534125d0ba033762b62db93ae2f81c71a543b6694df33773ebfa308R939-R976

devanbenz · 2026-06-23T17:12:44Z

run benchmark arrow-select

adriangbot · 2026-06-23T17:12:46Z

Hi @devanbenz, thanks for the request (#10136 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: Dandandan, Fokko, Jefffrey, Omega359, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas.

File an issue against this benchmark runner

alamb · 2026-06-23T18:10:35Z

run benchmark arrow-select

adriangbot · 2026-06-23T18:13:38Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4782118681-633-b8j47 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing db/10098/bmi-null-bool-filter (76c7efc) to 1ba5d48 (merge-base) diff
BENCH_NAME=arrow-select
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow-select
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-23T18:13:40Z

Benchmark for this request failed.

Last 20 lines of output:

Click to expand

    record_batch
    regexp_kernels
    row_format
    row_group_index_reader
    row_selection_cursor
    row_selector
    serde
    sort_kernel
    string_dictionary_builder
    string_run_builder
    string_run_iterator
    substring_kernels
    take_kernels
    union_array
    variant_builder
    variant_kernels
    variant_validation
    view_types
    writer_overhead
    zip_kernels

File an issue against this benchmark runner

alamb · 2026-06-23T21:05:04Z

run benchmark arrow_select

adriangbot · 2026-06-23T21:06:26Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4783458128-638-vs74c 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing db/10098/bmi-null-bool-filter (76c7efc) to 1ba5d48 (merge-base) diff
BENCH_NAME=arrow_select
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_select
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-23T21:06:28Z

Benchmark for this request failed.

Last 20 lines of output:

Click to expand

    record_batch
    regexp_kernels
    row_format
    row_group_index_reader
    row_selection_cursor
    row_selector
    serde
    sort_kernel
    string_dictionary_builder
    string_run_builder
    string_run_iterator
    substring_kernels
    take_kernels
    union_array
    variant_builder
    variant_kernels
    variant_validation
    view_types
    writer_overhead
    zip_kernels

File an issue against this benchmark runner

devanbenz · 2026-06-23T22:48:57Z

run benchmark arrow-select

@alamb my apologies, the bench is called filter_bits :P

alamb · 2026-06-24T19:26:28Z

run benchmark filter_bits

adriangbot · 2026-06-24T19:29:33Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4792893778-664-tn2q8 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing db/10098/bmi-null-bool-filter (76c7efc) to 1ba5d48 (merge-base) diff
BENCH_NAME=filter_bits
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_bits
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-24T19:32:33Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

New benchmark — branch-only results (no baseline comparison)

Details

group                                             db_10098_bmi-null-bool-filter
-----                                             -----------------------------
filter_bits (indices, kept 1/10)                  1.00     13.5±0.18µs        ? ?/sec
filter_bits (indices, kept 1/1024)                1.00   1014.5±3.55ns        ? ?/sec
filter_bits (indices, kept 1/2)                   1.00     77.3±0.43µs        ? ?/sec
filter_bits (slices, kept 1023/1024)              1.00      2.5±0.10µs        ? ?/sec
filter_bits (slices, kept 9/10)                   1.00     88.1±0.35µs        ? ?/sec
filter_bits optimized (indices, kept 1/10)        1.00     14.7±1.27µs        ? ?/sec
filter_bits optimized (indices, kept 1/1024)      1.00   225.8±12.39ns        ? ?/sec
filter_bits optimized (indices, kept 1/2)         1.00     73.9±6.47µs        ? ?/sec
filter_bits optimized (slices, kept 1023/1024)    1.00   1706.1±3.71ns        ? ?/sec
filter_bits optimized (slices, kept 9/10)         1.00     83.6±1.10µs        ? ?/sec
filter_bits sliced (indices, kept 1/10)           1.00     13.5±0.18µs        ? ?/sec
filter_bits sliced (indices, kept 1/1024)         1.00   1014.6±2.75ns        ? ?/sec
filter_bits sliced (indices, kept 1/2)            1.00     77.1±0.38µs        ? ?/sec
filter_bits sliced (slices, kept 1023/1024)       1.00      2.4±0.01µs        ? ?/sec
filter_bits sliced (slices, kept 9/10)            1.00     88.9±0.27µs        ? ?/sec

Resource Usage

branch

Metric	Value
Wall time	155.0s
Peak memory	6.8 MiB
Avg memory	5.2 MiB
CPU user	148.7s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

devanbenz · 2026-06-25T15:18:25Z

@alamb Could you please run the benchmark again but setting RUSTFLAGS="-C target-feature=-bmi2" to disable BMI on the host device? 🙏

alamb · 2026-06-29T14:49:53Z

run benchmark filter_bits

env:
RUSTFLAGS: "-C target-feature=-bmi2"

adriangbot · 2026-06-29T14:53:04Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4833873588-734-dldxz 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing db/10098/bmi-null-bool-filter (8e326b4) to 9f37683 (merge-base) diff
BENCH_NAME=filter_bits
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_bits
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-06-29T14:56:01Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

New benchmark — branch-only results (no baseline comparison)

Details

group                                             db_10098_bmi-null-bool-filter
-----                                             -----------------------------
filter_bits (indices, kept 1/10)                  1.00     13.5±0.17µs        ? ?/sec
filter_bits (indices, kept 1/1024)                1.00   1011.4±0.74ns        ? ?/sec
filter_bits (indices, kept 1/2)                   1.00     76.8±0.32µs        ? ?/sec
filter_bits (slices, kept 1023/1024)              1.00      2.5±0.00µs        ? ?/sec
filter_bits (slices, kept 9/10)                   1.00     95.1±0.30µs        ? ?/sec
filter_bits optimized (indices, kept 1/10)        1.00     14.3±1.07µs        ? ?/sec
filter_bits optimized (indices, kept 1/1024)      1.00    220.3±9.41ns        ? ?/sec
filter_bits optimized (indices, kept 1/2)         1.00     72.2±5.38µs        ? ?/sec
filter_bits optimized (slices, kept 1023/1024)    1.00   1707.5±3.10ns        ? ?/sec
filter_bits optimized (slices, kept 9/10)         1.00     62.6±0.70µs        ? ?/sec
filter_bits sliced (indices, kept 1/10)           1.00     13.5±0.18µs        ? ?/sec
filter_bits sliced (indices, kept 1/1024)         1.00   1011.5±0.76ns        ? ?/sec
filter_bits sliced (indices, kept 1/2)            1.00     76.7±0.38µs        ? ?/sec
filter_bits sliced (slices, kept 1023/1024)       1.00      2.4±0.00µs        ? ?/sec
filter_bits sliced (slices, kept 9/10)            1.00     96.1±0.31µs        ? ?/sec

Resource Usage

branch

Metric	Value
Wall time	145.0s
Peak memory	9.1 MiB
Avg memory	5.0 MiB
CPU user	141.8s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

devanbenz · 2026-06-30T14:41:29Z

@alamb Taking a look at the host machines. It doesn't look like the processor supports BMI2 intrinsics. So the benches are basically equal.

devanbenz · 2026-07-02T14:44:00Z

Now that #9848 Is merged in, I'm going to merge this PR with it and move the compress BMI method in to a shared arrow package. i.e. arrow-buffer.

alamb · 2026-07-02T18:33:48Z

nice

calculation a bit.

…nbenz/arrow-rs into db/10098/bmi-null-bool-filter

adjust testing and verify correctness between the two.

…nbenz/arrow-rs into db/10098/bmi-null-bool-filter

devanbenz added 2 commits June 12, 2026 07:46

feat: Working on BMI impl for filter_bits

3a64b9e

chore: use code from https://github.com/apache/arrow-rs/pull/9848/cha…

903f9c3

…nges#diff-f2fe5b601534125d0ba033762b62db93ae2f81c71a543b6694df33773ebfa308R939-R976

github-actions Bot added the arrow Changes to the arrow crate label Jun 12, 2026

devanbenz mentioned this pull request Jun 12, 2026

Optimize Arrow boolean/null filtering with BMI isntructions #10098

Open

devanbenz commented Jun 12, 2026

View reviewed changes

Comment thread arrow-buffer/src/util/bit_util.rs

fix: attempt to resolve regression for dense bitmaps

76c7efc

alamb mentioned this pull request Jun 23, 2026

feat(parquet): RowSelection can be backed by a BooleanBuffer #10141

Open

devanbenz added 2 commits June 25, 2026 10:46

feat: Merge branch 'master' into db/10098/bmi-null-bool-filter

27bf3b4

feat: simplify code a bit

8e326b4

This comment was marked as outdated.

Sign in to view

alamb mentioned this pull request Jun 30, 2026

feat(parquet): selective null padding for list child readers #9848

Merged

devanbenz added 2 commits July 2, 2026 09:44

Merge branch 'main' into db/10098/bmi-null-bool-filter

afed62b

feat: Moves compress from parquet workspace to arrow-buffer

5bbbdd8

github-actions Bot added the parquet Changes to the parquet crate label Jul 2, 2026

devanbenz added 2 commits July 2, 2026 12:10

chore: parquet fmt

9b54543

feat: Adds test for bmi2 path

a9e1587

alamb and others added 6 commits July 2, 2026 14:33

Merge branch 'main' into db/10098/bmi-null-bool-filter

57bebdd

feat: adds test for variable bit masks, and simplify the compress

3fa30ce

calculation a bit.

feat: Merge branch 'db/10098/bmi-null-bool-filter' of github.com:deva…

6b9b85a

…nbenz/arrow-rs into db/10098/bmi-null-bool-filter

Merge branch 'main' into db/10098/bmi-null-bool-filter

e672f0a

feat: seperate filter_bits to strat and compress

3cb8cce

adjust testing and verify correctness between the two.

feat: Merge branch 'db/10098/bmi-null-bool-filter' of github.com:deva…

499f9da

…nbenz/arrow-rs into db/10098/bmi-null-bool-filter

devanbenz marked this pull request as ready for review July 6, 2026 17:07

Uh oh!

Conversation

devanbenz commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

devanbenz commented Jun 23, 2026

Uh oh!

adriangbot commented Jun 23, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

adriangbot commented Jun 23, 2026

Uh oh!

adriangbot commented Jun 23, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

adriangbot commented Jun 23, 2026

Uh oh!

adriangbot commented Jun 23, 2026

Uh oh!

devanbenz commented Jun 23, 2026

Uh oh!

alamb commented Jun 24, 2026

Uh oh!

adriangbot commented Jun 24, 2026

Uh oh!

adriangbot commented Jun 24, 2026

Uh oh!

devanbenz commented Jun 25, 2026

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

alamb commented Jun 29, 2026

Uh oh!

adriangbot commented Jun 29, 2026

Uh oh!

adriangbot commented Jun 29, 2026

Uh oh!

devanbenz commented Jun 30, 2026

Uh oh!

devanbenz commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

devanbenz commented Jun 12, 2026 •

edited

Loading

devanbenz commented Jul 2, 2026 •

edited

Loading