Commits
Wes McKinney authored c9ac8696a33
ARROW-3212: [C++] Make IPC metadata deterministic, regardless of current stream position. Clean up stream / tensor alignment logic The detail of whether an `InputStream` or `OutputStream` is aligned to 8- or 64-byte boundary had contaminated a number of IPC functions, causing generated metadata (plus padding, if any) to be non-deterministic. This introduces a new `ipc::AlignStream` function and the desired use is to align the stream at the highest level possible. I've implemented the changes in the Python tensor serialization code paths. I removed various cruft where alignment issues leaked into the API This also resolves ARROW-2840, ARROW-2027, ARROW-2776, ARROW-1996 Author: Wes McKinney <wesm+git@apache.org> Author: Kouhei Sutou <kou@clear-code.com> Closes #2615 from wesm/ARROW-3212 and squashes the following commits: db23bfd01 <Wes McKinney> Use kTensorAlignment in more places bf105330a <Wes McKinney> Remove outdated comment, remove BufferOutputStream::Create convenience. Better document aligned SerializeTo tests 672ba9a2b <Kouhei Sutou> Update "Since" version f3fe30dab <Wes McKinney> Fix int conversion warning on windows 2bdb2fbf3 <Wes McKinney> Do not pad tensor message body. Align stream after writing each tensor in Serialize code path, fix tensorflow ops 06a3201fc <Wes McKinney> Remove position argument 11593434b <Wes McKinney> Actually check tensor 79d886909 <Wes McKinney> Update glib per ipc::ReadTensor API change 4cefedc09 <Wes McKinney> Place internal flatbuffer helper in header so does not need to be exported in DLL c4debf16d <Wes McKinney> Include warning in gcc7 70910df08 <Wes McKinney> Add unit test to verify that ARROW-1996 is fixed f30ddecbd <Wes McKinney> Do not write message body bytes if not included in the metadata 6a5d1d3ee <Wes McKinney> More fixes, assertions. Fix Python unit tests a987e398c <Wes McKinney> Add comments. tensor alignment test failing 9a392b76b <Wes McKinney> Add InputStream::Advance, fix Python compilation 5ad084f1e <Wes McKinney> Finish alignment refactor, fix C++ unit tests 47fc688e0 <Wes McKinney> Some refactoring of stream alignment