I have a web app hosted in Azure (Location: West US2). It is a backend API app and provide APIs for my front-side web app. In one of the APIs, I need to get the page content from a given URL. The codes like below:

using HttpClient httpClient = new HttpClient();
httpClient.Timeout = TimeSpan.FromMinutes(5);
var htmlContent = await httpClient.GetStringAsync(data.Url);
var browsingContext = BrowsingContext.New(Configuration.Default);
var browsingDocument = await browsingContext.OpenAsync(req => req.Content(htmlContent));
var textContent = browsingDocument.Body.TextContent;

But when I tried to access a URL like “https://www.allstate.com/auto-insurance/car-coverage-policies”, it threw Timeout error.

I tried some other URLs, like https://www.youtube.com, https://www.google.com, and they all worked. It looks my app can’t access some URLs.

So, does anyone know how to resolve this issue to make my app could access that ” “https://www.allstate.com/…” page? Do I need to add some web app configurations in Azure or add Azure virtual network?
I don’t know much about Azure network. it would be appreciated if anyone could provide detailed answers. Thanks a lot!