I am running subdomain enumeration using different tools where I will iterate a list of domains over a list of tools. To speed up the process, I implemented multi threading using concurrent.futures.ThreadPoolExecutor. In my scenario, I will have two ThreadPoolExecutor, first one is when I iterate through (lets say) 10000 domains, the second one will be when I iterate and run the 7 different command line tools(using subprocess to run the command) on one specific domain.

Inside the main.py

import enumFunctions
domains = get_domains() # a list of strings
count=0
with concurrent.futures.ThreadPoolExecutor(max_workers=50) as executor:
    for domain in domains:
        count+=1
        print(count)
        executor.submit(enumFunctions.subdomainEnum, domain[0],client)

inside enumFunctions.py is:

function_and_parameters = [
        (tool1,domain) # tool1 is a function
        (tool2,domain) # tool2 is a function
             ...
]
with concurrent.futures.ThreadPoolExecutor(max_workers=7) as executor:
    for tool,param in function_and_parameters:
        executor.submit(tool, param)

The expected output will be:

1
2
3
4
5
...
50 # stuck here for a while and waiting for worker to be freed to continue
51
52

However, what I am getting it

1

2
3
4
5
...
50 # after a few seconds/mins, the count immediately goes to 4000 something, instead of incrementing slowly.

Really appreciate if anyone could tell me where I did wrong here
Thanks in advance!