Commits


Antoine Pitrou authored and GitHub committed 3095344d68a
GH-40279: [C++] Reduce S3Client initialization time (#40299) ### Rationale for this change By default, S3Client instantiation is extremely slow (around 1ms for every instance). Investigation led to the conclusion that most of this time was spent inside the AWS SDK, parsing a hardcoded piece of JSON data when instantiating a AWS rule engine. Python benchmarks show this repeated initiatlization cost: ```python >>> from pyarrow.fs import S3FileSystem >>> %time s = S3FileSystem() CPU times: user 21.1 ms, sys: 0 ns, total: 21.1 ms Wall time: 20.9 ms >>> %time s = S3FileSystem() CPU times: user 2.37 ms, sys: 0 ns, total: 2.37 ms Wall time: 2.18 ms >>> %time s = S3FileSystem() CPU times: user 2.42 ms, sys: 0 ns, total: 2.42 ms Wall time: 2.23 ms >>> %timeit s = S3FileSystem() 1.28 ms ± 4.03 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) >>> %timeit s = S3FileSystem() 1.28 ms ± 2.6 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) >>> %timeit s = S3FileSystem(anonymous=True) 1.26 ms ± 2.46 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) ``` ### What changes are included in this PR? Instead of letting the AWS SDK create a new S3EndpointProvider for each S3Client, arrange to only create a single S3EndpointProvider per set of endpoint configuration options. This lets the 1ms instantiation cost be paid only when a new set of endpoint configuration options is given. Python benchmarks show the initialization cost has become a one-time cost: ```python >>> from pyarrow.fs import S3FileSystem >>> %time s = S3FileSystem() CPU times: user 20 ms, sys: 0 ns, total: 20 ms Wall time: 19.8 ms >>> %time s = S3FileSystem() CPU times: user 404 µs, sys: 49 µs, total: 453 µs Wall time: 266 µs >>> %time s = S3FileSystem() CPU times: user 361 µs, sys: 42 µs, total: 403 µs Wall time: 249 µs >>> %timeit s = S3FileSystem() 50.4 µs ± 227 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) >>> %timeit s = S3FileSystem(anonymous=True) 33.5 µs ± 306 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) ``` ### Are these changes tested? By existing tests. ### Are there any user-facing changes? No. * GitHub Issue: #40279 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>