Commits
Heres, Daniel authored and Andrew Lamb committed 066c6c3e3c1
ARROW-12136: [Rust][DataFusion] Reduce default batch_size to 8192 I did some comparisons with different batch sized with TCP-H on SF=1 in memory / 16 partitions. We chose a higher batch_size earlier as DF had some problems with smaller batch sizes (in hash join, but also because it missed the CoalesceBatches node). Smaller batch sizes have some performance benefit: more (possible) parallelism for smaller tables, batches are available sooner (think e.g. of `CoalesceBatches` with highly selective filter) and therefore more parallelism, etc. Also, memory usage can be reduced. Currently it seems a value around 8000 is the sweet spot for SF=1, only query 1 is faster with a slightly smaller batch size.  Closes #9834 from Dandandan/batch_size2 Authored-by: Heres, Daniel <danielheres@gmail.com> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>