I’d like some help with running Selenium on remote EC2 instances. My infrastructure uses a messaging system to execute tasks, such as Selenium. These are executed by ubuntu. The example Python script it uses is shown here:

from selenium.webdriver.chrome.service import Service
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options

if __name__ == '__main__':
    chrome_options = Options()
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--headless=new")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--window-size=1920,1080")

    service = Service('/usr/bin/chromedriver')
    driver = Chrome(service=service, options=chrome_options)
    driver.get("https://example.com/")
    print('I found a webpage title! %s' % driver.title)
    driver.quit()

And this script works very nicely! As long as there is an SSH connection active. To check for this, netstat -tan | grep ':22' can be executed to check for any (established) connections. When the Python script is run every 10 seconds, it reports whether it was successful or if it ran into the selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status: 1 error.

I can’t figure out why this works only when a SSH connection is active, and it remains working if VS Code is used to connect to this machine. If a plain ssh ubuntu@{IP} was used on the terminal, it will generate the WebDriverException error again the moment the SSH connection is closed. Any help is much appreciated, the PATH, env and DISPLAY variables are identical with and without connections. This happens on Ubuntu 22.04 LTS EC2 machines, running Python 3.10.12 and Selenium 4.22.0.