Commits

Weston Pace authored cf50b31904c
ARROW-14191: [C++][Dataset] Dataset writes should respect backpressure This PR adds the backpressure feature (introduced in ARROW-13611) to the dataset write node (introduced in ARROW-13542). To note: The python test here is unfortunately a bit slow for two reasons. First, we don't make the "how many rows can the dataset writer hold before backpressure applies" option a configurable option (it's not clear in what circumstances a user would ever change it) and it defaults to 64Mi rows. Second, there is no signal sent from the C++ side when backpressure has been applied. So we are forced to poll and guess when it seems we have stopped reading from the source. I'm open to suggestions (e.g. don't include the test, make the test a large memory test or something so it doesn't always run, any ideas we can use to test this better, etc.) Closes #11286 from westonpace/feature/ARROW-13611--scanning-datasets-backpressure-with-write Authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Weston Pace <weston.pace@gmail.com>