I am using a Blazor web app (server side) and have published it in a Linux VM on Microsoft Azure. The application runs well, but keeps crashing. After a long search for the source of the problem, I noticed that the app always crashes when it is crawled. In other words, when there are many requests per second. The app then no longer responds to new requests and loads endlessly for all new requests.

What can I do here? I have already seen that there is a LoadBalancer from Azure or a DDos Protection. However, I want search engines to be able to continue to search and index my site. Or is there a way to prevent this directly in the application and block the IP address?