I’m using Selenium (Python) to scrape data from Yahoo Finance. When using WebDriverWait to find the Close price on a given date from this page, I get an exception like this:

 Message: 
Stacktrace:
0   chromedriver                        0x0000000104c92940 chromedriver + 4368704
1   chromedriver                        0x0000000104c8add4 chromedriver + 4337108
2   chromedriver                        0x00000001048aec04 chromedriver + 289796
3   chromedriver                        0x00000001048f0e00 chromedriver + 560640
4   chromedriver                        0x00000001049295ec chromedriver + 792044
5   chromedriver                        0x00000001048e5ab4 chromedriver + 514740
6   chromedriver                        0x00000001048e650c chromedriver + 517388
7   chromedriver                        0x0000000104c56e5c chromedriver + 4124252
8   chromedriver                        0x0000000104c5bc4c chromedriver + 4144204
9   chromedriver                        0x0000000104c3c824 chromedriver + 4016164
10  chromedriver                        0x0000000104c5c57c chromedriver + 4146556
11  chromedriver                        0x0000000104c2e2d8 chromedriver + 3957464
12  chromedriver                        0x0000000104c7bec4 chromedriver + 4275908
13  chromedriver                        0x0000000104c7c040 chromedriver + 4276288
14  chromedriver                        0x0000000104c8aa34 chromedriver + 4336180
15  libsystem_pthread.dylib             0x00000001858b2f94 _pthread_start + 136
16  libsystem_pthread.dylib             0x00000001858add34 thread_start + 8

This doesn’t crash the script completely, but no data is collected. I can’t post my full code here, but this only occurs when looking for this specific element:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


driver = webdriver.Chrome()
url = f"https://finance.yahoo.com/quote/AMZN/history"
driver.get(url)

date = "Feb 29, 2024"

try:
    price_element = WebDriverWait(driver, 120).until(
        EC.presence_of_element_located((By.XPATH, f"//table[@class='table svelte-ewueuo']/tbody/tr[td[text()='{date}']]/td[count(//table[@class='table svelte-ewueuo']/thead/tr/th[text()='Date']/preceding-sibling::th)+5]"))
    )
    price = price_element.text
    print(price)
except Exception as e:
    print(f"EXCEPTION: {e}")

driver.quit()

I’ve verified that all xpaths are correct, and when this occurs, the element is always visible in the Selenium browser window (both literally and in the HTML). Weirdly, I have no issues retrieving the data on the first 2 iterations (iterating through a list of tickers), but this always happens on the third iteration and for every iteration afterward.

I’m using macOS 14.4.1, Selenium 4.20, Python 3.9.1, and Version 124.0.6367.208 (Official Build) (arm64)