Doing some webscraping, I’m loading the main page and needing to save the oauth token. The page loads and it does a post request to get a JSON file and that post request will have the “authtoken” in its request header, which I need to save , the issue is the JSON might be loaded in 2 seconds or 8 seconds , so is there a WebDriverWait-until combo I can use to check when the JSON is loaded? EC.url_contains only checks the current main page, not the post request url, is there a EC function that checks driver.request.headers for keywords or checks the post request URLs? thanks
driver = webdriver.Chrome(options=chrome_options)
driver.get("http://www.somewebsite.com")
# to check when that json file is finished loading, but url_contains only refers to the url of the main page, not the smaller external post request urls
try:
WebDriverWait(driver, 10).until(EC.url_contains("www.jsonlink.com/json/"))
except TimeoutException:
print("timeout")
# this is how i find the auth token, but it works after everything is loaded..and i need to know when it's loaded before i run this part
for request in driver.requests:
if "www.jsonlink.com/json/" in request.url:
token = request.headers.get("authtoken")