I have used Selenium with Python for months to interact successfully with a website that disables direct HTTP requests to download files (these requests lead to 403 errors).

Instead, I used a headless browser to navigate to the page that hosts links to these files, collect the links and then navigate to those links, activating a download. This worked well … until the past few days.

The problem seems to be that I can no longer disable Selenium’s experimental option prompt_for_download. The browser now triggers a ‘Save As’ pop-up when the file begins to download, which interrupts the browser’s navigation.

This is the relevant part of my script, which had been working (my work environment is Windows 11):

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager

# headless browswing
chrome_options = Options()
chrome_options.add_argument("--headless")

# enabling browser downloads
chrome_options.add_experimental_option("prefs", {
    "download.default_directory": r"C:\my_download_path\",
    "download.prompt_for_download": False,
    "download.directory_upgrade": True,
    "safebrowsing.enabled": True
})

# activate browser instance    
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options
)

# skipping code here that navigates to the relevant page
# and identifies the URL of the file ('download_url')

# navigate to file URL to download it
driver.get(download_url)

# wait for download
time.sleep(5)
    
# continue processing

I’m keen to know whether others have faced similar problems recently, or whether there are obvious weaknesses in the code above.