Commits


Philipp Moritz authored and Robert Nishihara committed cecbcf7de00
ARROW-1744: [Plasma] Provide TensorFlow operator to transfer Tensors between Plasma and TensorFlow This is based off of #2046 from @pcmoritz and @concretevitamin, who wrote the original TensorFlow Op with help from @ericl. In addition to the previous PR, this supports serialization of arbitrary type tensors. To get it working, arrow has to be compiled in the following way: ``` cmake -DARROW_PLASMA=on -DARROW_TENSORFLOW=on -DARROW_PYTHON=on .. make -j8 sudo make install ``` And pyarrow like that: ``` PYARROW_WITH_PLASMA=1 PYARROW_WITH_TENSORFLOW=1 python setup.py develop ``` The PR also includes utilities that should be generally useful in converting between TensorFlow and Arrow. It is currently supposed to be used to transfer weights in and out of TensorFlow but hopefully in the future there will also be support for data Tensors. Author: Philipp Moritz <pcmoritz@gmail.com> Author: Peter Schafhalter <pschafhalter@berkeley.edu> Closes #2104 from pschafhalter/tensor-serialization and squashes the following commits: 50c6635 <Philipp Moritz> don't import tensorflow on manylinux1 3b7bbc7 <Philipp Moritz> skip tests that use tensorflow inside of manylinux1 container 097d48d <Philipp Moritz> Merge branch 'master' into tensor-serialization df44982 <Philipp Moritz> Merge branch 'master' into tensor-serialization 13e4db3 <Philipp Moritz> remove setting the deleted variable to nullptr b79c710 <Philipp Moritz> more fixes 200ad89 <Philipp Moritz> move build flags earlier b4b4134 <Philipp Moritz> tensorflow namespace b0731ad <Philipp Moritz> fixes 10e369a <Philipp Moritz> more fixes 01e580c <Philipp Moritz> more cleanups 027d683 <Philipp Moritz> fix e494f28 <Philipp Moritz> clean up code 1ef52d0 <Philipp Moritz> fix 90757c1 <Philipp Moritz> fix gpu code d5d3740 <Philipp Moritz> fix d137fcf <Philipp Moritz> compile pyarrow without cxx-11 ABI too 81f34ff <Philipp Moritz> fix b4e02e1 <Philipp Moritz> fix test 749e21e <Philipp Moritz> cleanups and test fb8786e <Philipp Moritz> fix bc3a5bb <Philipp Moritz> cleanups 56c9cf1 <Philipp Moritz> fix c6066ab <Philipp Moritz> fix 2fe3523 <Philipp Moritz> cleanups 4dc4ff3 <Philipp Moritz> fix TensorFlow test 3264efe <Philipp Moritz> test acc1f0b <Philipp Moritz> fix test 372551f <Philipp Moritz> pkg-config 64ee050 <Philipp Moritz> fix test d83b9ae <Philipp Moritz> fix test 2385e7c <Philipp Moritz> fix af4eb2f <Philipp Moritz> cleanups 6a34556 <Philipp Moritz> move things around afa5dba <Philipp Moritz> fix b29bf39 <Philipp Moritz> linting 14c9e50 <Philipp Moritz> fix linting eb1a576 <Philipp Moritz> fix 47a21c9 <Philipp Moritz> add license 1c8c5fa <Philipp Moritz> Merge branch 'master' into tensor-serialization 1442be0 <Philipp Moritz> test more dtypes ead1a2e <Philipp Moritz> test af68371 <Philipp Moritz> debug2 a2131ad <Philipp Moritz> debug 531f4a7 <Philipp Moritz> fix be83a20 <Philipp Moritz> try datatype 077266b <Philipp Moritz> fixes 1164d5a <Philipp Moritz> fixes 4f641c3 <Philipp Moritz> fix warnings a674720 <Philipp Moritz> fix 140a2b9 <Philipp Moritz> needs debugging 8b1dd8c <Philipp Moritz> get write path working d7ead2b <Philipp Moritz> fix d291f64 <Philipp Moritz> fix f461b38 <Philipp Moritz> fix linting 3b01abe <Philipp Moritz> fix 232ed6f <Philipp Moritz> add tensor utilities f7466de <Peter Schafhalter> Formatting 3ed436b <Peter Schafhalter> Formatting f4ad8e5 <Peter Schafhalter> Check that all tensors have the same dtype f601bb0 <Peter Schafhalter> Convert Tensor type to Arrow 4167d1d <Peter Schafhalter> Fix tensor serialization e261f2c <Philipp Moritz> fixes for the op d381ff3 <Philipp Moritz> fixes c705d53 <Philipp Moritz> fix for python 2.7 22b5564 <Philipp Moritz> fix wheel building ae2e7ca <Philipp Moritz> adapt code for arrow f6cef78 <Philipp Moritz> add initial tensorflow op