arrow

ARROW-8487: [FlightRPC] Provide a way to target a particular payload size

(Recreated since I rebased before reopening.)

I'd appreciate feedback on this. This is a specialized API to limit the payload size when a client is sending data to a server.

While Flight by default removes the gRPC-level limits on message sizes, we've found setting server-side limits on the size of incoming messages is still useful to ensure clients don't overwhelm a server/exhaust server memory by sending very large messages. However, if you exceed the limit at the gRPC level, the server resets the connection, making it hard for a client to recover gracefully. This PR adds an optional client-side check, so the client can catch the error and retry with a smaller record batch, or provide a meaningful error to the user if the batch cannot be made small enough.

Unlike solutions involving pa.ipc.get_record_batch_size, this avoids duplicate re-serialization of the record batch.

Closes #7398 from lidavidm/arrow-8487

Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>