Commits


Sam Albers authored and Neal Richardson committed d2cbe9e0e2c
ARROW-16144: [R] Write compressed data streams (particularly over S3) This PR enables reading/writing compressed data streams over s3 and locally and adds some tests to test some of those round trips. For the filesystem path I had to do a little regex on the string for compression detection but any feedback on alternative approaches is very welcome. Previously supplying a file with a compression extension wrote out an uncompressed file. Here is a reprex of the updated writing behaviour: ```r library(arrow, warn.conflicts = FALSE) ## local write_csv_arrow(mtcars, file = file) write_csv_arrow(mtcars, file = comp_file) file.size(file) [1] 1303 file.size(comp_file) [1] 567 ## or with s3 dir <- tempfile() dir.create(dir) subdir <- file.path(dir, "bucket") dir.create(subdir) minio_server <- processx::process$new("minio", args = c("server", dir), supervise = TRUE) Sys.sleep(2) stopifnot(minio_server$is_alive()) s3_uri <- "s3://minioadmin:minioadmin@?scheme=http&endpoint_override=localhost%3A9000" bucket <- s3_bucket(s3_uri) write_csv_arrow(mtcars, bucket$path("bucket/data.csv.gz")) write_csv_arrow(mtcars, bucket$path("bucket/data.csv")) file.size(file.path(subdir, "data.csv.gz")) [1] 567 file.size(file.path(subdir, "data.csv")) [1] 1303 ``` Closes #13183 from boshek/ARROW-16144 Lead-authored-by: Sam Albers <sam.albers@gmail.com> Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>