I am trying to recursively fetch files and their metadata from dir and subdir using asyncio in python.

async def get_files_from_site(site_id):

    async def process_drive(drive_data):
        # Fetch and process files for each drive


    async def process_drive_files(drive_id):
        # Create and gather tasks to process each item (file or folder)


    async def process_item(item, drive_id):
        # If item is a folder, process the folder
        # If item is a file, append permissions


    async def process_folder(folder, drive_id):
        # Get children of the folder
        # Create and gather tasks to process each child item
        # Ensure each folder's items are processed in parallel


    try:
        drives = await fetch_drives()
        tasks = [asyncio.create_task(process_drive(drive)) for drive in drives]
        await asyncio.gather(*tasks)

# Call get_files_from_site with a specific site_id

This is the output of the folder process log:

T: 2024-04-29 19:30:37.826112   |   L:0   |   Processing: Folder_M6ZJI
T: 2024-04-29 19:30:38.147987   |   L:0   |   Processing: Folder_SGF5M
T: 2024-04-29 19:30:38.436990   |   L:0   |   Processing: Folder_0ENXT
T: 2024-04-29 19:30:38.707225   |   L:1   |   Processing: Folder_0THFP
T: 2024-04-29 19:30:38.983634   |   L:1   |   Processing: Folder_RUK33
T: 2024-04-29 19:30:39.280125   |   L:1   |   Processing: Folder_VWJDI
T: 2024-04-29 19:30:39.973852   |   L:1   |   Processing: Folder_VCYEO
T: 2024-04-29 19:30:40.277223   |   L:1   |   Processing: Folder_X36WJ
T: 2024-04-29 19:30:40.621078   |   L:1   |   Processing: Folder_EJSCO
T: 2024-04-29 19:30:40.921566   |   L:1   |   Processing: Folder_RV0EA
T: 2024-04-29 19:30:41.210761   |   L:1   |   Processing: Folder_T08LD
T: 2024-04-29 19:30:41.663906   |   L:1   |   Processing: Folder_3ZQ3K
T: 2024-04-29 19:30:41.960160   |   L:1   |   Processing: Folder_OWU0X
T: 2024-04-29 19:30:42.231661   |   L:1   |   Processing: Folder_VS5YA
T: 2024-04-29 19:30:42.613695   |   L:1   |   Processing: Folder_QNNEC
T: 2024-04-29 19:30:43.497759   |   L:1   |   Processing: Folder_NU61A
T: 2024-04-29 19:30:43.820132   |   L:1   |   Processing: Folder_4PI59
T: 2024-04-29 19:30:44.172989   |   L:1   |   Processing: Folder_7QX7K
T: 2024-04-29 19:30:44.517731   |   L:1   |   Processing: Folder_38CCT
T: 2024-04-29 19:30:54.745630   |   L:2   |   Processing: Folder_EHYSV
T: 2024-04-29 19:30:59.844896   |   L:2   |   Processing: Folder_F50A6
T: 2024-04-29 19:31:15.852660   |   L:2   |   Processing: Folder_RZ41F
T: 2024-04-29 19:31:16.124211   |   L:2   |   Processing: Folder_5MP5F
T: 2024-04-29 19:31:16.417750   |   L:2   |   Processing: Folder_ENXFY
T: 2024-04-29 19:31:16.736579   |   L:2   |   Processing: Folder_T0QAB
T: 2024-04-29 19:31:26.511858   |   L:2   |   Processing: Folder_77NAT
T: 2024-04-29 19:31:26.829356   |   L:2   |   Processing: Folder_QZIMB
T: 2024-04-29 19:31:27.197568   |   L:2   |   Processing: Folder_8D9BU
T: 2024-04-29 19:31:27.520321   |   L:2   |   Processing: Folder_GITP3
T: 2024-04-29 19:31:36.704139   |   L:2   |   Processing: Folder_EBN51
T: 2024-04-29 19:31:37.010878   |   L:2   |   Processing: Folder_446HW
T: 2024-04-29 19:31:37.292827   |   L:2   |   Processing: Folder_2GE3H
T: 2024-04-29 19:31:37.596673   |   L:2   |   Processing: Folder_MWCL1
T: 2024-04-29 19:31:37.900958   |   L:2   |   Processing: Folder_3MKX4
T: 2024-04-29 19:31:38.205139   |   L:2   |   Processing: Folder_QLW7M
T: 2024-04-29 19:31:38.562049   |   L:2   |   Processing: Folder_2AXAR
T: 2024-04-29 19:31:38.845413   |   L:2   |   Processing: Folder_GRGAD
T: 2024-04-29 19:31:39.140256   |   L:2   |   Processing: Folder_VQ270
T: 2024-04-29 19:31:39.451358   |   L:2   |   Processing: Folder_HT0EE
T: 2024-04-29 19:31:39.753310   |   L:2   |   Processing: Folder_NNCYW
T: 2024-04-29 19:31:40.033072   |   L:2   |   Processing: Folder_M4VJX
T: 2024-04-29 19:31:40.327013   |   L:2   |   Processing: Folder_3GXIB
T: 2024-04-29 19:31:40.631007   |   L:2   |   Processing: Folder_J9ZUW
T: 2024-04-29 19:31:40.924172   |   L:2   |   Processing: Folder_M8WIU
T: 2024-04-29 19:31:41.237224   |   L:2   |   Processing: Folder_4J26A
T: 2024-04-29 19:31:41.546810   |   L:2   |   Processing: Folder_Z43EQ
T: 2024-04-29 19:31:41.833033   |   L:2   |   Processing: Folder_OVV8Q
T: 2024-04-29 19:31:42.129396   |   L:2   |   Processing: Folder_9SI2N
T: 2024-04-29 19:31:42.388558   |   L:2   |   Processing: Folder_5TC72
T: 2024-04-29 19:31:42.669539   |   L:2   |   Processing: Folder_C4XUW
T: 2024-04-29 19:31:42.954231   |   L:2   |   Processing: Folder_2D66B
T: 2024-04-29 19:31:43.261566   |   L:2   |   Processing: Folder_ITD3E
T: 2024-04-29 19:31:43.553835   |   L:2   |   Processing: Folder_VO6O9
T: 2024-04-29 19:31:43.848590   |   L:2   |   Processing: Folder_ORUGH
T: 2024-04-29 19:31:44.142270   |   L:2   |   Processing: Folder_KER1S
T: 2024-04-29 19:31:44.400799   |   L:2   |   Processing: Folder_65OS7
T: 2024-04-29 19:31:44.713195   |   L:2   |   Processing: Folder_ELZED
T: 2024-04-29 19:31:45.066785   |   L:2   |   Processing: Folder_76DLS
T: 2024-04-29 19:31:45.510208   |   L:2   |   Processing: Folder_4TR7J
T: 2024-04-29 19:31:46.007586   |   L:2   |   Processing: Folder_8UI3H
T: 2024-04-29 19:31:46.274339   |   L:2   |   Processing: Folder_ZCNOK
T: 2024-04-29 19:31:46.581570   |   L:2   |   Processing: Folder_3E8EJ
T: 2024-04-29 19:31:46.858867   |   L:2   |   Processing: Folder_ZWI77
T: 2024-04-29 19:31:47.138109   |   L:2   |   Processing: Folder_ZDM2E
T: 2024-04-29 19:31:47.413728   |   L:2   |   Processing: Folder_H4BXZ
T: 2024-04-29 19:31:47.687955   |   L:2   |   Processing: Folder_71WIE
T: 2024-04-29 19:31:47.958552   |   L:2   |   Processing: Folder_XKK1W
T: 2024-04-29 19:31:48.250405   |   L:2   |   Processing: Folder_DHT11
T: 2024-04-29 19:31:48.504886   |   L:2   |   Processing: Folder_LK84T
T: 2024-04-29 19:31:48.774206   |   L:2   |   Processing: Folder_0I49S
T: 2024-04-29 19:31:49.065718   |   L:2   |   Processing: Folder_JMOR6
T: 2024-04-29 19:31:49.343031   |   L:2   |   Processing: Folder_JWW21
T: 2024-04-29 19:31:49.677100   |   L:2   |   Processing: Folder_VVQLY
T: 2024-04-29 19:31:49.972921   |   L:2   |   Processing: Folder_M0JUL
T: 2024-04-29 19:31:50.259652   |   L:2   |   Processing: Folder_4V8NE
T: 2024-04-29 19:31:50.550460   |   L:2   |   Processing: Folder_EV73D
T: 2024-04-29 19:31:50.824147   |   L:2   |   Processing: Folder_J5Z5N
T: 2024-04-29 19:31:51.119320   |   L:2   |   Processing: Folder_MD5FM
T: 2024-04-29 19:31:51.396102   |   L:2   |   Processing: Folder_FIGDN
T: 2024-04-29 19:31:51.735500   |   L:2   |   Processing: Folder_U0L3D
T: 2024-04-29 19:31:52.025084   |   L:2   |   Processing: Folder_UXIST
T: 2024-04-29 19:32:00.562895   |   L:2   |   Processing: Folder_JTTLK
T: 2024-04-29 19:32:00.863096   |   L:2   |   Processing: Folder_H1NDK
T: 2024-04-29 19:32:01.187330   |   L:2   |   Processing: Folder_183UN
T: 2024-04-29 19:32:01.465031   |   L:2   |   Processing: Folder_NF1OR
T: 2024-04-29 19:32:01.743344   |   L:2   |   Processing: Folder_394FL
T: 2024-04-29 19:32:02.016046   |   L:2   |   Processing: Folder_LOS9M
T: 2024-04-29 19:32:02.298562   |   L:2   |   Processing: Folder_0O2ZR
T: 2024-04-29 19:32:02.674472   |   L:2   |   Processing: Folder_77GG4
T: 2024-04-29 19:32:03.021415   |   L:2   |   Processing: Folder_GP3AC
T: 2024-04-29 19:32:03.327592   |   L:2   |   Processing: Folder_JB0Q7
T: 2024-04-29 19:32:32.551194   |   L:3   |   Processing: Folder_A7K4R
T: 2024-04-29 19:32:32.859409   |   L:3   |   Processing: Folder_FV7UL
T: 2024-04-29 19:32:33.153646   |   L:3   |   Processing: Folder_R94F6
T: 2024-04-29 19:32:33.427088   |   L:3   |   Processing: Folder_QAX0V
T: 2024-04-29 19:32:33.703515   |   L:3   |   Processing: Folder_0FBNY

My intuition is that all folders in a depth level N should be called at the same time. I am able to acheive such parallelism using node in single process. Is it acheivable in python what am i doing wrong here?

Tried ayncio.gather(*tasks) but doesnt seem like all call go out at the same time.