This is my usecase:
Traverse a filesystem, gauge its entire size and then upload it to Dropbox.
Looks quite easy and is quite easy. Now if I do it using celery and spawn a thread for each folder(With sub files), then it becomes an easier process to gather all the data and gauge the size and then upload it to dropbox.
But I feel I’m doing something wrong here. If I implement this on a server, I’m surely overloading it by spawning a thread for each folder that is out there. Hence, what algorithms should I use to make this a faster process? Both as a data retrieval and data uploading.
Links/References would help.
6
The short version: absent unusual circumstances, you should use a single-threaded traversal. As @CodesInChaos says, your computer is likely to be I/O bound in both phases of your task: size-checking and uploading alike.
Trying to spawn a thread per directory is a recipe for fork-bombing your machine into oblivion. More precisely, there is no point in invoking more threads than are necessary to saturate your performance bottlenecks. Note that there are 2 likely bottlenecks: the performance of your filesystem, and the performance of your network link to Dropbox.
For uploading data to Dropbox, your network is almost certainly the bottleneck. You should easily be able to saturate any broadband link with a single-threaded traversal.
For finding the size of the filesystem, your bottleneck will also be I/O, but the details depend on the actual filesystem. For a solid-state drive, you probably won’t benefit from more than a single thread. I suspect that you may gain a modest benefit from a bounded number of threads in a single-disk filesystem, as the OS can potentially schedule head stops a bit more efficiently if it has a queue. The question is, would a small performance margin be worth the cost of implementing and maintaining a more complex implementation? (E.g., handling cases like multiple hard links may be difficult enough with a single-threaded implementation…)
A distributed filesystem might benefit quite a lot from a multithreaded traversal. However, the details of how to extract the best performance are likely to depend on the specific system.
1