I’m currently developing a React application that displays a deeply nested folder structure retrieved from AWS S3. My folder structure can contain 20k to 30k files, and I’m using MUI X Tree View to render the UI. However, I’m running into performance issues, especially when trying to manage and display such a large dataset.

Goals:

• Batch Data Retrieval: I want to implement a solution to fetch data
  in batches from S3 instead of loading everything at once, which is
  causing slowdowns.
• Virtualization: I aim to use virtualization to render only the
  visible parts of the tree for better performance as users scroll
  through the data.

Current Implementation:

• I am currently using the listObjectsV2 API from AWS S3 to fetch folder contents. However, loading all items at once leads to performance degradation.

Challenges :

• How can I effectively structure my S3 requests to batch data retrieval?

• What is the best way to handle loading more data as the user scrolls
  or expands folders, especially to maintain responsiveness in the UI?

Questions:

1. How can I implement batch data retrieval to load files dynamically while using MUI X Tree View?
2. What strategies can I use to ensure that the UI remains responsive
   and performs well when dealing with large datasets?
3. Are there best practices for combining virtualization with dynamic
   loading in MUI X Tree View?

I expected the application to display the folder structure quickly, allowing users to expand folders and see the contents without significant delays. Specifically, I thought that fetching data in smaller batches as users interacted with the tree would improve performance and provide a smoother experience.

Any insights or examples you could share would be greatly appreciated! Thank you for your help!