There is a website that I am trying to scrape. In order to do that, there are first several steps I must complete. After sending several requests, I am able to get to the point where I am able to enter a string in the search bar. All those requests give expected responses. It is only the request for searching that string or keyword in the search bar that gives unexpected response. The response is 200, but it is different from that I see in the response of the network tab on the browser. The difference starts from TargetLocation, which in the expected response (that sent from the server to the browser) is “noDisplay” but that I am getting is “viewManager” and it gives no result in the row_data.

what could I be doing wrong? I am implementing this in scrapy and using headers and payloads to send requests.