Commits


David Li authored and Joris Van den Bossche committed 65a4e65e354
ARROW-11596: [Python][Dataset] make ScanTask.execute() eager This changes 2 things: - ScanTask.execute() is now eagerly evaluated, so any work involved in creating a record batch reader is done up front. For example, a Parquet file will actually begin reading (but not decoding) data. This mirrors the behavior in C++. This also makes it easier to separate stages of work when pipelining reads (by manually dispatching scan tasks). - This has the side effect of working around a SIGSEGV in the Cython generator, caused because Cython raises StopIteration without holding the GIL when you have a generator that uses "with [no]gil". In my tests this is fixed on Cython master but not any stable or development release. Closes #9480 from lidavidm/arrow-11596 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>