Hoping to find my answers here. I’m using Selenium with Python to crawl data, but after about 2,500 records out of say 50,000, I encounter this error and my ECS Fargate instance terminates:

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
(session not created: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Here’s the relevant traceback:

File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
super().__init__(command_executor=executor, options=options)
File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 292, in start_session
response = self.execute(Command.NEW_SESSION, caps)["value"]
File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
(session not created: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Here is what my task definition looks like:

{
    "containerDefinitions": [
        {
            "cpu": 0,
            "portMappings": [],
            "essential": true,
            "mountPoints": [],
            "volumesFrom": [],
            "systemControls": []
        }
    ],
    "networkMode": "awsvpc",
    "revision": 12,
    "volumes": [],
    "status": "ACTIVE",
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "2048",
    "memory": "4096",
    "tags": []
}