I’m attempting to open a large CSV file with the SmarterCSV rails gem but on my M3 Pro with 32 Gigs Ram it just hangs and fails to open the file (even if using chunks) when running:

SmarterCSV.process(open("the_file.csv"), {chunk_size: 10000, convert_values_to_numeric: false})

If I run this on my PC and open in rails (using WSL2) it opens in seconds without even using chunks… Any idea on what could be the bottleneck here? Any help identifying the bottleneck would be greatly appreciated.