arrow

ARROW-17306: [C++] Provide an optimized `GetFileInfoGenerator` specialization for `LocalFileSystem` (#13796)

Introduce a specialization of `GetFileInfoGenerator` in the `LocalFileSystem` class.

This implementation tries to improves performance by hiding latencies at two levels:
1. Child directories can be readahead so that listing directories entries from disk can be achieved in parallel with other work;
2. Directory entries can be `stat`'ed and yielded in chunks so that the `FileInfoGenerator` consumer can start receiving entries before a large directory is fully processed.

Both mechanisms can be tuned using dedicated parameters in `LocalFileSystemOptions`.

Signed-off-by: Pavel Solodovnikov <pavel.al.solodovnikov@gmail.com>
Co-Authored-by: Igor Seliverstov <iseliverstov@querifylabs.com>

Lead-authored-by: Pavel Solodovnikov <pavel.al.solodovnikov@gmail.com>
Co-authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Antoine Pitrou <antoine@python.org>