Public
  1. Public

arrow

Public

Commits

David LiWes McKinney
be8f9413b8a
David Li authored and Wes McKinney committed d90c1597ce4
ARROW-4858: [Flight/Python] enable FlightDataStream to be implemented in Python

This enables use cases like converting and streaming a large Pandas DataFrame in chunks, or generating/processing data on the fly.

It could be simplified if we decide to only support iterators of RecordBatches (versus right now where it accepts Tables, RecordBatches, other FlightDataStreams, etc.)

Author: David Li <David.M.Li@twosigma.com>

Closes #3890 from lihalite/arrow-py-data and squashes the following commits:

ed108cfd5 <David Li> Document that Flight/Python helpers require GIL to construct
0bcd8e144 <David Li> Check that PyGeneratorFlightDataStream returns consistent schema
27298945e <David Li> Enable FlightDataStream to be implemented in Python