arrow

ARROW-300: [Format] Proposal for "trivial" IPC body buffer compression using either LZ4 or ZSTD codecs

This permits each constituent buffer forming the IPC message body to be compressed independently using either the LZ4 or ZSTD codecs. The compressed data is written with a 64-bit signed integer prefix indicating the size of the uncompressed data.

Note that there is still interest in the project in developing metadata for data-type-specific encoding and compression on the wire, but there is still more design work to do there.

As a special case, the size prefix may be "-1" to indicate that the buffer is uncompressed, which can be helpful for small buffers or buffers where the producer decided that the compression ratio was not beneficial.

Note that this change is not forward compatible, so "old" readers will not be able to read compressed messages nor determine that they cannot be read. If a "new" writer writes uncompressed messages, there is no issue. Since we are still pre-1.0 I think this is OK. We may consider increasing the Flatbuffer metadata version to prevent old readers from blindly treating compressed buffers as uncompressed.

Closes #6707 from wesm/ipc-body-buffer-compression

Authored-by: Wes McKinney <wesm+git@apache.org>
Signed-off-by: Wes McKinney <wesm+git@apache.org>