I am making a program in python which take in a large list and use each item of the list to perform a task.
One way to make this efficient is to split the list in multiple parts and then use multi threading on those sub-lists to reduce the time of execution.
My question is how we can dynamically create threads (sub processes) depending upon how much CPU is being used (other apps might also be running in the background). For example if there are no background processes running, I want my script to dynamically create maximum available sub processes (threads) and reduce the execution time to a great extent, and if there are other task running in the background, it should create fewer number of sub processes.
We can use multi threading in python by manually creating threads, but I want it to happen dynamically, which should also take in account how much cpu does other applications are using and make more threads accordingly.