arrow

ARROW-530: [C++/Python] Provide subpools for better memory allocation …

…tracking

Currently we can only track the amount of bytes allocated by the main memory pool or the alternative jemalloc implementation. To better understand certain situation, we should provide a MemoryPool proxy implementation that tracks only the amount of memory that was made through its direct calls but delegates the actual allocation to an underlying pool.

Authors: Rok Mihevc <rok@mihevc.org> and Alex Hagerman <alex@unexpectedeof.net>

Usage example:
---------------------
import pyarrow as pa

def report(pool, proxy_pool):
    print("Total: ", pa.total_allocated_bytes())
    print("Default pool: ", pool.bytes_allocated())
    print("Proxy allocated: ", proxy_pool.proxy_bytes_allocated())

pool = pa.default_memory_pool()
proxy_pool = pa.ProxyMemoryPool(pool)

report(pool, proxy_pool)
a1 = pa.array([0]*1000, memory_pool=pool)
report(pool, proxy_pool)
a2 = pa.array([0]*1, memory_pool=proxy_pool)
report(pool, proxy_pool)
a3 = pa.array([0]*1000, memory_pool=proxy_pool)
report(pool, proxy_pool)
a3 = pa.array([0]*1000, memory_pool=proxy_pool)
report(pool, proxy_pool)

Result:
---------------------
Total:  0
Default pool:  0
bytes_allocated: 0
Proxy allocated:  0
Total:  8128
Default pool:  8128
bytes_allocated: 0
Proxy allocated:  0
Allocate: size = 64
Allocate: size = 256
Reallocate: old_size = 256 - new_size = 64 - proxy_allocated - 128
Total:  8256
Default pool:  8256
bytes_allocated: 128
Proxy allocated:  128
Allocate: size = 128
Allocate: size = 8192
Reallocate: old_size = 8192 - new_size = 8000 - proxy_allocated - 8256
Total:  16384
Default pool:  16384
bytes_allocated: 8256
Proxy allocated:  8256
Allocate: size = 128
Allocate: size = 8192
Reallocate: old_size = 8192 - new_size = 8000 - proxy_allocated - 16384
Free: size = 128
Free: size = 8000
Total:  16384
Default pool:  16384
bytes_allocated: 8256
Proxy allocated:  8256

Author: Rok <rok@mihevc.org>

Closes #2057 from rok/master and squashes the following commits:

eabd8a4f <Rok> changing ProxyMemoryPool instantiation
96e58ee6 <Rok> fixing linting issue
5bebf97d <Rok> adding docstring
195a1367 <Rok> implementing feedback
0645e500 <Rok> refactoring ProxyMemoryPool interface
17a5c9b6 <Rok> ARROW-530 - C++ linter fix
f90b8970 <Rok> C++ unit tests for ARROW-530 passing
7c4b7692 <Rok> adding c++ unit tests
62dcdb0a <Rok> ARROW-530: C++/Python: Provide subpools for better memory allocation tracking