Commits


Wes McKinney authored and Antoine Pitrou committed e4bf4297585
ARROW-6994: [C++] Fix aggressive RSS inflation on macOS when jemalloc background_thread is not enabled In investigating ARROW-6994 and ARROW-7305 I was shocked to find the following behavior from this test script https://gist.github.com/wesm/193f644d10b5aee8c258b8f4f81c5161 (requires data file attached to ARROW-7305) this is what I see both on master and 0.15.1 ``` $ python arrow7305.py Starting RSS: 67297280 Read CSV RSS: 179224576 Wrote Parquet RSS: 645177344 Waited 1 second RSS: 645177344 Read CSV RSS: 703504384 Wrote Parquet RSS: 707674112 Waited 1 second RSS: 707674112 ... Waited 1 second RSS: 1147301888 ``` This contrasts with Linux where RSS stabilizes around 170MB at the end. The macOS behavior on 0.14.1 is slightly better, though RSS ends up at ~465MB after the script runs. When the background thread option is disabled this patch sets the decay time for unused memory pages to 0 so they are released immediately to the OS. Without understanding more about what's going on the current behavior on master this seems better than releasing again what is currently there. With this patch I have ``` $ python arrow7305.py Starting RSS: 68505600 Read CSV RSS: 179671040 Wrote Parquet RSS: 288759808 Waited 1 second RSS: 288759808 Read CSV RSS: 298987520 Wrote Parquet RSS: 301359104 Waited 1 second RSS: 301359104 Read CSV RSS: 308961280 Wrote Parquet RSS: 313081856 Waited 1 second RSS: 313081856 ... Read CSV RSS: 315908096 Wrote Parquet RSS: 315822080 Waited 1 second RSS: 315822080 Read CSV RSS: 315822080 Wrote Parquet RSS: 315944960 Waited 1 second RSS: 315944960 Waited 1 second RSS: 315944960 ``` Closes #6100 from wesm/ARROW-6994 and squashes the following commits: 5af6c203b <Antoine Pitrou> Add comment c2e2f6017 <Wes McKinney> clang-format 5c1b0b1a3 <Wes McKinney> Set decay_ms to 0 if background_thread is unavailable Lead-authored-by: Wes McKinney <wesm+git@apache.org> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>