Commits


Heres, Daniel authored and Andrew Lamb committed a1ac15f208f
ARROW-11018: [Rust][DataFusion] Add support for column-level statistics, null count. This adds extra statistics on the amount of nulls per column. This is a step towards supporting more flexible cost based optimizations. Second step is adding number distinct, min, max values. With that we can have a good estimate of selectivity of filters, supporting more cases in which we could apply optimizations such as reordering joins. Notes * It is a `Vec` to be in line with things like projections * I think in memory / parquet (locally) it could be provided on read, for other datasources it might be useful to have support for something like `ANALYZE TABLE`. FYI @andygrove this is one step for more flexible CBO for joins combined with filters. Closes #8998 from Dandandan/null_count Lead-authored-by: Heres, Daniel <danielheres@gmail.com> Co-authored-by: Daniël Heres <danielheres@gmail.com> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>