I’m trying to download a zipped csv-file, unzip it and parse it. Unfortunately, it seems like the unzipped file is too big (almost 1GB) to be converted to a string. What’s the best way to split the data? Is it possible to create a stream that streams the Uint8array to a string which is then streamed and read by the csv reader (papaparse)?
This is my current approach:
const response = await fetch(url);
const buffer = await response.arrayBuffer();
const int8Array = new Uint8Array(buffer);
const unzipped = fflate.unzipSync(int8Array);
const csvString = fflate.strFromU8(Object.values(unzipped)[0]); // this throws an error
const readCSV = Papa.parse(csvString, {
header: true,
delimiter: ',',
skipEmptyLines: true,
});
Thanks for any help!