I have a TypeScript function that tries to download a large number of small documents concurrently. Here’s the code:
const bulkDownload = async (analyses: FullAnalysesResponse[]) => {
setIsLoading(true);
const promises = analyses.map(async (analysis) => {
const documentInfo = await getDocumentInfo({ documentId: analysis.downloadFileId! });
const attachment = await downloadDocument({ documentId: analysis.downloadFileId! });
FileSaver.saveAs(new File([attachment], documentInfo.name, { type: documentInfo.mimeType }));
});
const results = await Promise.allSettled(promises);
setIsLoading(false);
results.forEach((result, index) => {
const analysisSerialNumber = analyses[index].deviceSerialNumber;
result.status === 'fulfilled'
? successfulQueries.push(analysisSerialNumber)
: failedQueries.push(analysisSerialNumber);
});
return { failedQueries, successfulQueries };
};
The issue is that when I trigger this function to download multiple files at once, not all the files are downloaded. The number of downloaded files changes every time I run the function, and I never get all the files. All the API calls are working, so all the promises are successful. The issue seems to come from the FileSaver.saveAs
function.
I also tried a version that uses a simple for...of
loop, which works fine:
const bulkDownload = async (analyses: FullAnalysesResponse[]) => {
setIsLoading(true);
const successfulQueries: string[] = [];
const failedQueries: string[] = [];
for (const analysis of analyses) {
try {
const documentInfo = await getDocumentInfo({ documentId: analysis.downloadFileId! });
const attachment = await downloadDocument({ documentId: analysis.downloadFileId! });
FileSaver.saveAs(new File([attachment], documentInfo.name, { type: documentInfo.mimeType }));
successfulQueries.push(analysis.deviceSerialNumber);
} catch (error) {
failedQueries.push(analysis.deviceSerialNumber);
}
}
setIsLoading(false);
return { failedQueries, successfulQueries };
};
The for...of
version works reliably but is slower since it downloads the files sequentially. I would like to understand why the first (concurrent) function is not working as expected. I assumed that running the downloads concurrently would be more efficient.
Any insights on why this happens, and how to fix it while keeping the performance benefits of concurrent downloads?
18
I’m hesitant to answer this, because I fear there may be an element of “magic number” about it
What you want is to control the number of concurrent file downloads (i.e. not the API requests, since they seem to work fine as fast as you can request them), but this is not exactly possible, since FileSaver.js
can’t tell you when a file has completed downloading. However, maybe with some tweaking of the “magic numbers” you can get a consistent (faster than one at a time) result.
So, the following code contains the allSettledConcurrent
function that combines your .map
and Promise.allSettled
into one function
In case you’re wondering, I wrote this function, along with a suite of others (allConcurrent
, throttleAllConcurrent
, throttleAllSettledConcurrent
, throttlePromise
etc) years ago for various use cases
// MAGIC numbers
const afterDownloadDelay = 100;
const concurrency = 10;
async function allSettledConcurrent(items, concurrency, fn) {
const results = new Array(items.length);
const queue = items.map((item, index) => ({ item, index }));
const doFn = async ({ item, index }) => {
try {
const value = await fn(item);
results[index] = { value, status: "fulfilled" };
} catch (reason) {
results[index] = { reason, status: "rejected" };
}
return queue.length && doFn(queue.shift());
};
const slots = queue.splice(0, concurrency).map(doFn);
await Promise.all(slots);
return results;
}
const bulkDownload = async (analyses: FullAnalysesResponse[]) => {
setIsLoading(true);
const results = await allSettledConcurrent(analyses, concurrency, async (analysis) => {
const documentInfo = await getDocumentInfo({ documentId: analysis.downloadFileId! });
const attachment = await downloadDocument({ documentId: analysis.downloadFileId! });
FileSaver.saveAs(new File([attachment], documentInfo.name, { type: documentInfo.mimeType }));
if (afterDownloadDelay) {
await new Promise(resolve => setTimeout(resolve, afterDownloadDelay));
}
});
setIsLoading(false);
results.forEach((result, index) => {
const analysisSerialNumber = analyses[index].deviceSerialNumber;
result.status === 'fulfilled'
? successfulQueries.push(analysisSerialNumber)
: failedQueries.push(analysisSerialNumber);
});
return { failedQueries, successfulQueries };
};