I’m using docker compose in a VM with only two containers, one is an api (flask) that uses selenium and the other one is the docker image: selenium/standalone-chrome:latest. The problem is when i docker compose up locally it runs just fine hitting localhost:5000 for example the api starts the scraping without any issue.

But when i try to do it on the virtual machina using the ipv4:port (ex: 132.45.22.110:5000) this huge error appears:

    deploy_argy-flask-app-1            | INFO:werkzeug:45.70.185.230 - - [28/May/2024 15:31:18] "GET /start HTTP/1.1" 200 -
    deploy_argy-flask-app-1            | /app/argy.py:132: DeprecationWarning: desired_capabilities has been deprecated, please pass in an Options object with options kwarg
    deploy_argy-flask-app-1            |   driver = webdriver.Remote(
    deploy_argy-flask-app-1            | WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ebd65b995b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /wd/hub/session
    deploy_argy-flask-app-1            | WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ebd65b996d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /wd/hub/session
    deploy_argy-flask-app-1            | WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ebd65b99820>: Failed to establish a new connection: [Errno 111] Connection refused')': /wd/hub/session
    deploy_argy-flask-app-1            | Exception in thread Thread-3:
    deploy_argy-flask-app-1            | Traceback (most recent call last):
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 198, in _new_conn
    deploy_argy-flask-app-1            |     sock = connection.create_connection(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    deploy_argy-flask-app-1            |     raise err
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection
    deploy_argy-flask-app-1            |     sock.connect(sa)
    deploy_argy-flask-app-1            | ConnectionRefusedError: [Errno 111] Connection refused
    deploy_argy-flask-app-1            |
    deploy_argy-flask-app-1            | The above exception was the direct cause of the following exception:
    deploy_argy-flask-app-1            |
    deploy_argy-flask-app-1            | Traceback (most recent call last):
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    deploy_argy-flask-app-1            |     response = self._make_request(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    deploy_argy-flask-app-1            |     conn.request(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 400, in request
    deploy_argy-flask-app-1            |     self.endheaders()
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
    deploy_argy-flask-app-1            |     self._send_output(message_body, encode_chunked=encode_chunked)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
    deploy_argy-flask-app-1            |     self.send(msg)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/http/client.py", line 951, in send
    deploy_argy-flask-app-1            |     self.connect()
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 238, in connect
    deploy_argy-flask-app-1            |     self.sock = self._new_conn()
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 213, in _new_conn
    deploy_argy-flask-app-1            |     raise NewConnectionError(
    deploy_argy-flask-app-1            | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ebd65b999d0>: Failed to establish a new connection: [Errno 111] Connection refused
    deploy_argy-flask-app-1            |
    deploy_argy-flask-app-1            | The above exception was the direct cause of the following exception:
    deploy_argy-flask-app-1            |
    deploy_argy-flask-app-1            | Traceback (most recent call last):
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    deploy_argy-flask-app-1            |     self.run()
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/threading.py", line 870, in run
    deploy_argy-flask-app-1            |     self._target(*self._args, **self._kwargs)
    deploy_argy-flask-app-1            |   File "/app/argy.py", line 132, in scraping_loop
    deploy_argy-flask-app-1            |     driver = webdriver.Remote(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
    deploy_argy-flask-app-1            |     self.start_session(capabilities, browser_profile)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
    deploy_argy-flask-app-1            |     response = self.execute(Command.NEW_SESSION, parameters)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 438, in execute
    deploy_argy-flask-app-1            |     response = self.command_executor.execute(driver_command, params)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/remote_connection.py", line 290, in execute
    deploy_argy-flask-app-1            |     return self._request(command_info[0], url, body=data)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/remote/remote_connection.py", line 311, in _request
    deploy_argy-flask-app-1            |     response = self._conn.request(method, url, body=body, headers=headers)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/_request_methods.py", line 144, in request
    deploy_argy-flask-app-1            |     return self.request_encode_body(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/_request_methods.py", line 279, in request_encode_body
    deploy_argy-flask-app-1            |     return self.urlopen(method, url, **extra_kw)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/poolmanager.py", line 444, in urlopen
    deploy_argy-flask-app-1            |     response = conn.urlopen(method, u.request_uri, **kw)
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    deploy_argy-flask-app-1            |     return self.urlopen(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    deploy_argy-flask-app-1            |     return self.urlopen(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    deploy_argy-flask-app-1            |     return self.urlopen(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    deploy_argy-flask-app-1            |     retries = retries.increment(
    deploy_argy-flask-app-1            |   File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 515, in increment
    deploy_argy-flask-app-1            |     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    deploy_argy-flask-app-1            | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4444): Max retries exceeded with url: /wd/hub/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ebd65b999d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

I think it might be an issue with how i set up my webdriver remote so here goes some information:

docker-compose.yml :

    version: '3.8'
    
    services:
      flask-app:
        build: .
        ports:
          - "5000:5000"
        depends_on:
          - selenium-standalone
        environment:
          - FLASK_ENV=development
        env_file:
          - .env
        networks:
          - my-network
      
      selenium-standalone:
        image: selenium/standalone-chrome:latest
        ports:
          - "4444:4444"
          - "7900:7900"
        shm_size: 2gb
        privileged: true
        networks:
          - my-network
    
    networks:
      my-network:
        driver: bridge

selenium config in the script:

    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("privileged")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-software-rasterizer")  
    chrome_options.add_argument("--window-size=1920,1080")
    
    driver = webdriver.Remote(
            command_executor='http://selenium-standalone:4444/wd/hub',
            options=chrome_options)

flask config:

    if __name__ == "__main__":
        app.run(host="0.0.0.0", port=5000, debug=True)

If it helps, this is the message the console shows after the docker compose up :

     ⠿ Network deploy_argy_my-network               Created                                                                                                                     0.2s
     ⠿ Container deploy_argy-selenium-standalone-1  Created                                                                                                                     0.1s
     ⠿ Container deploy_argy-flask-app-1            Created                                                                                                                     0.1s
    Attaching to deploy_argy-flask-app-1, deploy_argy-selenium-standalone-1
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:55,973 INFO Included extra file "/etc/supervisor/conf.d/chrome-cleanup.conf" during parsing
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:55,973 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:55,984 INFO RPC interface 'supervisor' initialized
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:55,985 CRIT Server 'unix_http_server' running without any HTTP authentication checking
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:55,987 INFO supervisord started with pid 8
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:56,994 INFO spawned: 'xvfb' with pid 9
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,006 INFO spawned: 'vnc' with pid 10
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,029 INFO spawned: 'novnc' with pid 11
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,046 INFO spawned: 'selenium-standalone' with pid 13
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,096 INFO success: xvfb entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,099 INFO success: vnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,100 INFO success: novnc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
    deploy_argy-selenium-standalone-1  | 2024-05-28 15:30:57,101 INFO success: selenium-standalone entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
    deploy_argy-selenium-standalone-1  | Appending Selenium options: --heartbeat-period 30
    deploy_argy-selenium-standalone-1  | Appending Selenium options: --reject-unsupported-caps false
    deploy_argy-selenium-standalone-1  | Setting up SE_NODE_GRID_URL...
    deploy_argy-selenium-standalone-1  | Selenium Grid Standalone configuration:
    deploy_argy-selenium-standalone-1  | [network]
    deploy_argy-selenium-standalone-1  | relax-checks = true
    deploy_argy-selenium-standalone-1  |
    deploy_argy-selenium-standalone-1  | [node]
    deploy_argy-selenium-standalone-1  | session-timeout = "300"
    deploy_argy-selenium-standalone-1  | override-max-sessions = false
    deploy_argy-selenium-standalone-1  | detect-drivers = false
    deploy_argy-selenium-standalone-1  | drain-after-session-count = 0
    deploy_argy-selenium-standalone-1  | max-sessions = 1
    deploy_argy-selenium-standalone-1  |
    deploy_argy-selenium-standalone-1  | [[node.driver-configuration]]
    deploy_argy-selenium-standalone-1  | display-name = "chrome"
    deploy_argy-selenium-standalone-1  | stereotype = '{"browserName": "chrome", "browserVersion": "125.0", "platformName": "Linux", "goog:chromeOptions": {"binary": "/usr/bin/google-chrome"}}'
    deploy_argy-selenium-standalone-1  | max-sessions = 1
    deploy_argy-selenium-standalone-1  |
    deploy_argy-selenium-standalone-1  | Starting Selenium Grid Standalone...
    deploy_argy-selenium-standalone-1  | Tracing is disabled
    deploy_argy-flask-app-1            |  * Serving Flask app 'argy'
    deploy_argy-flask-app-1            |  * Debug mode: on
    deploy_argy-flask-app-1            | INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
    deploy_argy-flask-app-1            |  * Running on all addresses (0.0.0.0)
    deploy_argy-flask-app-1            |  * Running on http://127.0.0.1:5000
    deploy_argy-flask-app-1            |  * Running on http://172.29.0.3:5000
    deploy_argy-flask-app-1            | INFO:werkzeug:Press CTRL+C to quit
    deploy_argy-flask-app-1            | INFO:werkzeug: * Restarting with stat
    deploy_argy-selenium-standalone-1  | 15:31:01.681 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
    deploy_argy-selenium-standalone-1  | 15:31:01.747 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
    deploy_argy-flask-app-1            | WARNING:werkzeug: * Debugger is active!
    deploy_argy-flask-app-1            | INFO:werkzeug: * Debugger PIN: 217-542-865
    deploy_argy-selenium-standalone-1  | 15:31:04.335 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors
    deploy_argy-selenium-standalone-1  | 15:31:04.470 INFO [NodeOptions.report] - Adding chrome for {"browserName": "chrome","browserVersion": "125.0","goog:chromeOptions": {"binary": "u002fusru002fbinu002fgoogle-chrome"},"platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
    deploy_argy-selenium-standalone-1  | 15:31:04.539 INFO [Node.<init>] - Binding additional locator mechanisms: relative
    deploy_argy-selenium-standalone-1  | 15:31:04.587 INFO [GridModel.setAvailability] - Switching Node 4bf6fab8-bd65-4d37-be9d-68ba288a0c5b (uri: http://172.29.0.2:4444) from DOWN to UP
    deploy_argy-selenium-standalone-1  | 15:31:04.588 INFO [LocalDistributor.add] - Added node 4bf6fab8-bd65-4d37-be9d-68ba288a0c5b at http://172.29.0.2:4444. Health check every 120s
    deploy_argy-selenium-standalone-1  | 15:31:04.889 INFO [Standalone.execute] - Started Selenium Standalone 4.21.0 (revision 79ed462ef4): http://172.29.0.2:4444

I’ve tried changing the url in command_executor='http://selenium-standalone:4444/wd/hub' but nothing worked so far. Thanks