Commits


Kenta Murata authored and Kouhei Sutou committed aab4946ad28
ARROW-4506: [Ruby] Add Arrow::RecordBatch#raw_records I want to add Arrow::RecordBatch#raw_records method to convert a record batch object to a nested array. This is the first step to implement the feature. The following things are out of scope of this pull-request. - Conversion of half-float values to Ruby's float. - Unit treatment of Time32 and Time64 - Conversion of the following compound data types to Ruby objects: - ListType - StructType - UnionType - DictionaryType ### TODO: - [x] Extracting raw values of HalfFloatArray - [x] Extracting ListArray - [x] Extracting StructArray - [x] Extracting SparseUnionArray - [x] Extracting DenseUnionArray - [x] FixedSizeBinary - [x] Date32 - [x] Date64 - [x] Timestamp - [x] Decimal128 - [x] Struct - [x] Dictionary - [x] Extracting indices of DictionaryArray - [x] Make CI passed - [x] Add benchmark script Author: Kenta Murata <mrkn@mrkn.jp> Author: Kouhei Sutou <kou@clear-code.com> Closes #3587 from mrkn/raw_records and squashes the following commits: 00197e4b <Kouhei Sutou> Split test files 0d0d5170 <Kouhei Sutou> Replace a large StructArray test with small tests 61f9774b <Kouhei Sutou> Use {"field_name" => value} for union value 5fa9ba03 <Kouhei Sutou> Finish replacing tests for StructArray 83bf55a3 <Kouhei Sutou> Add support for nested StructArray c065a8a2 <Kouhei Sutou> Add tests for StructArray#raw_records 795c96a8 <Kouhei Sutou> Add support for "_" for data type name 87ab55da <Kouhei Sutou> Add support for nil value as NULL for struct field value 08b85c81 <Kouhei Sutou> Add support for nested list 4d78da49 <Kouhei Sutou> Remove resolved TODO 024506f1 <Kouhei Sutou> Remove needless tests e30a9e73 <Kouhei Sutou> Add support for nil in ListArrayBuilder#append_values 55066950 <Kouhei Sutou> Add primitive array tests 67808a2b <Kouhei Sutou> Add support for building BinaryArray 44779252 <Kouhei Sutou> Add tests for primitive arrays cb3cb471 <Kouhei Sutou> Add support for NullArray 728b82c9 <Kouhei Sutou> Reduce scope 021dbee2 <Kouhei Sutou> Use .cpp for C++ 4d01f85b <Kouhei Sutou> Use constexpr 7f5d6cc3 <Kouhei Sutou> Fix style db1e1c25 <Kouhei Sutou> Remove needless reference 028e3c4a <Kouhei Sutou> Use auto ce67b5bb <Kouhei Sutou> Stop reusing block argument name 750afff5 <Kouhei Sutou> Use .cpp for C++ extension a2032754 <Kouhei Sutou> Fold a long line 46d63aeb <Kouhei Sutou> Use auto 03233bd5 <Kouhei Sutou> Use Red Arrow in build directly 3a776306 <Kouhei Sutou> Fix package name for MSYS2 022dd073 <Kenta Murata> Fix the benchmark for dictionary array 3e18e85f <Kenta Murata> Rename a benchmark file 2c6142c8 <Kenta Murata> Rename a directory 40595617 <Kenta Murata> Fix benchmark task 5d380fdc <Kenta Murata> Use values between 2**16 and 2**32-1 for testing UInt32Array d0b8d0fc <Kenta Murata> Fix styling f41550d0 <Kenta Murata> Stop using precomputed scales of time unit bdf7090e <Kenta Murata> Remove needless scope blocks 8e81e9b2 <Kouhei Sutou> Implement converters based on visitor 31ca243a <Kenta Murata> Add extension files in the gem package ec11062b <Kenta Murata> Add arrow ext dir in $LOAD_PATH d0a4733b <Kenta Murata> Fix benchmark against the removal of convert_decimal: option 24d14800 <Kenta Murata> Guard RVAL2GOBJ by rb::protect 73063288 <Kenta Murata> Drop convert_decimal: option ba96767a <Kenta Murata> Remove a needless member function dfa6b4ae <Kenta Murata> Introduce require_extension_library method to load arrow.so ee8dcccf <Kenta Murata> Avoid using rb_str_new_cstr 7e51f7a8 <Kenta Murata> Fix variable names in test 8d1c36e4 <Kenta Murata> Use #pragma once bba7eb4e <Kenta Murata> Rename a variable b47fbfeb <Kenta Murata> Use GOBJ2RVAL_UNREF correctly 4a73d5eb <Kenta Murata> Use auto instead of VALUE d59d72be <Kenta Murata> Put some codes out side of rb::protect block 9995dec9 <Kenta Murata> Use rb_enc_str_new with rb_ascii8bit_encoding for binary string creation e0b26563 <Kenta Murata> Replace assert with DCHECK 70ccc327 <Kenta Murata> Make cArrowRecordBatch a local variable 16cbe8d9 <Kenta Murata> Use rb::RawMethod d6b7d3da <Kenta Murata> Remove needless require ed507393 <Kenta Murata> Fix styling da2c49f6 <Kenta Murata> Rename rb_cDate to cDate 4645c17c <Kenta Murata> Rename cRecordBatch to cArrowRecordBatch 2c410cf6 <Kenta Murata> Remove namespace comments 55df564d <Kenta Murata> Rename files 0a4016b3 <Kenta Murata> Replace license headers 180c7c3e <Kenta Murata> Use static timestamp_range in benchmark 1413aa6b <Kouhei Sutou> Set PKG_CONFIG_PATH to build Red Arrow 509af8ff <Kenta Murata> Add benchmark task e3fa4e62 <Kenta Murata> Fix word usage 31c8a270 <Kenta Murata> Use double quotations 9568a371 <Kenta Murata> Remove redundant sub test cases 7fb64a9b <Kenta Murata> Remove parentheses with empty argument 5bdd9952 <Kenta Murata> Remove needless require 56634502 <Kenta Murata> Remove arrow_ruby_compile function 794c7a27 <Kenta Murata> Revert needless changes be026ed1 <Kouhei Sutou> Fix style c0b7442d <Kouhei Sutou> Add "compile" task 99442745 <Kouhei Sutou> Add support "rake clean" and "rake clobber" 56bbc15b <Kouhei Sutou> Run extconf.rb automatically in test/run-test.rb 4d9440d0 <Kouhei Sutou> Use Ext++ d16b6e1d <Kouhei Sutou> Sort alphabetically 6217e862 <Kouhei Sutou> Add support for auto package install 08818907 <Kouhei Sutou> Remove rake-compiler dependency 38d6f530 <Kenta Murata> Make the default value of conver_decimal true f3cd6724 <Kenta Murata> Move gem entries from Gemfile into gemspec 2b97d3e3 <Kenta Murata> Fix benchmarks 0a9f9ece <Kenta Murata> Fix the random state of Faker in benchmark 26c7ea17 <Kenta Murata> Add benchmark scripts 7eaa0690 <Kenta Murata> Separate raw_records test de943457 <Kenta Murata> Fix missing const modifiers 40c4966c <Kenta Murata> Support Struct in Union cba224a3 <Kenta Murata> Support Dictionary in Union 95517e3a <Kenta Murata> Add tests for dense union in dense union 86f8bc11 <Kenta Murata> Fix travis script 82ce1bcb <Kenta Murata> Add license comment 9df5765a <Kenta Murata> Refactoring test 5016c6a9 <Kenta Murata> Support Dictionary in UnionArray 462480d6 <Kenta Murata> Support Date32, Date64, and Timestamp in UnionArray 0d358a3c <Kenta Murata> Refactoring 5a2276bf <Kenta Murata> Use non-default field name for a list in a record batch 6b9de32d <Kenta Murata> Add support of FixedSizeBinary in Union 786e4388 <Kenta Murata> Refactoring of Decimal128 converter fcd3ab95 <Kenta Murata> Support SparseUnion ce10bcbd <Kenta Murata> Support Decimal128 in DenseUnion 5b14d106 <Kenta Murata> Add partial support of DenseUnion 432f05ae <Kenta Murata> Fix encoding bug 94fe1c48 <Kenta Murata> Add tentative support of HalfFloat 72a64f5e <Kenta Murata> Refactoring 61f0bc50 <Kenta Murata> Support Dictionary indices 63876209 <Kenta Murata> Support Struct 223db821 <Kenta Murata> Use RETURN_NOT_OK cead59bb <Kenta Murata> Save errinfo if rb::error created from state 0186066c <Kenta Murata> Extract ArrayConverter class 294c7496 <Kenta Murata> Supply PKG_CONFIG_PATH to rake compile 53fa1ffd <Kenta Murata> Fix CI script 17a69c2a <Kenta Murata> Support List 88119d5a <Kenta Murata> Remove pure-Ruby version aa52a174 <Kenta Murata> Tweak comment and error message 6c8a2e2c <Kenta Murata> Add tentative supports of Time32 and Time64 d33d77d2 <Kenta Murata> Support Timestamp 72935bc9 <Kenta Murata> Support Date32 and Date64 93518946 <Kenta Murata> Use rb_jump_tag to raise deferred exception 476f544b <Kenta Murata> Add convert_decimal kwarg 54e59e78 <Kenta Murata> Fix VisitValue for nil 5f7a78ca <Kenta Murata> Use RawRecordsBuilder d8d54d67 <Kenta Murata> Add RawRecordsBuilder 4d6d6392 <Kenta Murata> Update test case 8d68d5bc <Kenta Murata> Add a partial native implementation of RecordBatch#raw_records 25a1925c <Kenta Murata> Add test and tentative implementation of RecordBatch#raw_records