Commits


Andy Grove authored and Wes McKinney committed 075e4dd311f
ARROW-9452: [Rust] [DataFusion] Optimize ParquetScanExec This PR changes the way channels are used to communicate with the parquet scan thread. Rather than the caller send a request per batch, the thread just pushes batches to a bounded channel. This is far more efficient as shown by the almost 2x improvement in the benchmark crate. Before: ``` Running benchmarks with the following options: Opt { debug: false, iterations: 3, batch_size: 4096, path: "/mnt/nyctaxi/parquet/year=2019", file_format: "parquet" } Executing 'fare_amt_by_passenger' Query 'fare_amt_by_passenger' iteration 0 took 2267 ms Query 'fare_amt_by_passenger' iteration 1 took 2221 ms Query 'fare_amt_by_passenger' iteration 2 took 2224 ms ``` After: ``` Running benchmarks with the following options: Opt { debug: false, iterations: 3, batch_size: 4096, path: "/mnt/nyctaxi/parquet/year=2019", file_format: "parquet" } Executing 'fare_amt_by_passenger' Query 'fare_amt_by_passenger' iteration 0 took 1233 ms Query 'fare_amt_by_passenger' iteration 1 took 1203 ms Query 'fare_amt_by_passenger' iteration 2 took 1221 ms ``` Closes #7743 from andygrove/ARROW-9452 Authored-by: Andy Grove <andygrove73@gmail.com> Signed-off-by: Wes McKinney <wesm@apache.org>