Public
  1. Public

arrow

Public

Commits

Igor SuhorukovGitHub
a2f3666d792
Igor Suhorukov authored and GitHub committed 78351cec5af
ARROW-17303: [Java][Dataset] Read Arrow IPC files by NativeDatasetFactory (#13760) (#13811)

This PR allow developers to create Dataset from ARROW IPC files in JVM code like:
`FileSystemDatasetFactory factory = new FileSystemDatasetFactory(rootAllocator(), NativeMemoryPool.getDefault(),
            FileFormat.ARROW_IPC, arrowDatasetURL);`

It is foundation for Apache Spark arrow data source to process huge existing partitioned datasets in ARROW file format without additional data format conversion

Lead-authored-by: Igor Suhorukov <igor.suhorukov@gmail.com>
Co-authored-by: igor.suhorukov <igor.suhorukov@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>