Commits


Will Jones authored and GitHub committed c72f84a48b4
ARROW-15906: [C++][Python][R] By default, don't create or delete S3 buckets (#13206) BREAKING CHANGE: modifies `S3FileSystem` to not allow creating or deleting buckets by default. Two new options are added: `allow_bucket_creations`, which enables creating buckets, and `allow_bucket_deletion`, which enables deleting buckets. Outside of tests, most use cases will not want to create or delete buckets, and doing so accidentally is not desirable either. Buckets have governance controls like permissions and cost-tracking tags. To make it easy for users to transition, the `FromUri` method also supports these arguments: ```python from pyarrow.fs import FileSystem uri = "s3://minioadmin:minioadmin@?scheme=http&endpoint_override=localhost%3A9000" fs, path = FileSystem.from_uri(uri) fs.create_dir("test") # Traceback (most recent call last): # File "<stdin>", line 1, in <module> # File "pyarrow/_fs.pyx", line 463, in pyarrow._fs.FileSystem.create_dir # check_status(self.fs.CreateDir(directory, recursive=recursive)) # File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status # raise IOError(message) # OSError: Bucket 'test' not found. To create buckets, enable the allow_bucket_creation option. uri = uri + "&allow_bucket_creation=True&allow_bucket_deletion=True" fs, path = FileSystem.from_uri(uri) fs.create_dir("test") fs.delete_dir("test") ``` ```r fs <- FileSystem$from_uri("s3://minioadmin:minioadmin@?scheme=http&endpoint_override=localhost%3A9000")$fs fs$CreateDir("test") #> Error: IOError: Bucket 'test' not found. To create buckets, enable the allow_bucket_creation option. fs <- FileSystem$from_uri( paste0("s3://minioadmin:minioadmin@?scheme=http&endpoint_override=localhost%3A9000", "&allow_bucket_creation=TRUE&allow_bucket_deletion=TRUE") )$fs fs$CreateDir("test") fs$DeleteDir("test") ``` Authored-by: Will Jones <willjones127@gmail.com> Signed-off-by: Antoine Pitrou <pitrou@free.fr>