I am trying to scrap information form a website (USTA) with Python but failed with 403 responses. The webpage doesn’t require any login and I can find the request and response with developper tools in a browser without any issue. For example, while access this url:
https://www.usta.com/en/home/play/player-search/profile.html#?uaid=2017185101
there was one request post to:
https://www.usta.com/usta/api?type=playerInfo
with header:
dicData= {"selection":{"uaid":2017185101},"output":{"ratings":"true", "wtn":"true", "extendedProfile":"true"}}
I am not sure why my code did not work but looks like there was a Hash header item added during the process via a browser. Any suggestion on solving this issue? Here is my code:
import requests
strURL="https://www.usta.com/usta/api?type=playerInfo"
dicData= {"selection":{"uaid":2017185101},"output":{"ratings":"true", "wtn":"true", "extendedProfile":"true"}}
dicHead = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0", "Accept": "*/*", "Accept-Language": "en-US,en;q=0.5", "Content-Type": "application/json; charset=utf-8"}
r = requests.post(strURL, dicData, headers=dicHead)
print(r)