TL;DR: I’m only able to connect via SSH tunnel to MySQL/Percona (in Docker) after previously connecting (with the same credentials) via Adminer.

Goal

I want to use a dockerized SSH Server to expose a port and then tunnel into other containers like MySQL internally. The idea is to use SSH to secure the connection.

Setup

I’m using the following Docker containers:

• linuxserver/openssh-server
• Percona MySQL Server
• Adminer (formerly phpMinAdmin), only used for testing currently

The SSH server is exposed to the internet and shares the Docker network with the MySQL Server. In addition, Adminer is exposed for testing currently:

  Exposed to outside                    
     │                                  
     │       Server/Docker Containers   
     │      ┌───────────────────────┐   
     │   ┌──┼─────────────────────┐ │   
     └►  │  │  SSH Server/Tunnel  │ │   
         └──┼───────────────────┬─┘ │   
            │ ┌─────────────────▼─┐ │   
            │ │ MySQL Server      │ │   
            │ └─────────────────▲─┘ │   
         ┌──┼───────────────────┴─┐ │   
     ┌─► │  │  Adminer/PHPMyAdmin │ │   
     │   └──┼─────────────────────┘ │   
     │      └───────────────────────┘   
     └─── Only exposed temporarily for testing

The Problem

For some reason, I’m only able to login to MySQL (via tunnel) from outside after a successful login via Adminer. This is reproducable. Here is what I’m doing:

• On the server start Docker containers:
  • docker compose up to start SSH container
  • docker compose up to start MySQL Server (+ Adminer) containers
• Use MySQL client HeidiSQL (from my local computer) to login.
  • Receive error: Access denied for user 'root'@'172.18.0.2' (using password: YES)
• Open Adminer in my web browser (exposed to the internet).
  • Login, Server: percona-mysql-server, User: root, Password: THE_PASSWORD
  • Works, Adminer shows that I’m logged in as [email protected]
• Now again use my MySQL client to login.
  • Works! WTF?

Now, what I can figure from this is that the SSH tunnel is likely not the problem as I’m receiving the Access denied for user error message. Previously (when the tunnel was not setup properly) I would get network errors instead of this MySQL error. I also tried adding other users via a MySQL init script. For these users I could reproduce the same problem: Only after one successful login via Adminer, I was able to login via the tunnel.

So maybe Adminer is running some initialization that is currently missing from my setup? Some cache clearing or something?

Configuration files

Here are the configuration files I’m using:

MySQL

Here is my docker-compose.yml for the MySQL server. I also tried to use an init script to bind-address to 0.0.0.0 earlier as well as setting up other users, but that also did not change anything. The network was earlier created via docker network create tunnel

services:
  percona-mysql-server:
    image: percona:8.0
    environment:
      MYSQL_ROOT_PASSWORD: THE_PASSWORD
    expose:
      - "3306"
    volumes:
      - ./percona-data:/var/lib/mysql
    networks:
      - tunnel

  adminer:
    image: adminer
    ports:
      - 8080:8080
    networks:
      - tunnel

networks:
  tunnel:
    external: true

SSH Tunnel

I’ve greatly simplified the tunnel docker-compose.yml for this question and only included the network part as I’m rather sure this is working (as I’m not getting network errors but MySQL errors). Most important, the values AllowAgentForwarding, AllowTcpForwarding, GatewayPorts, X11Forwarding are enabled and I’m connecting via authorized_keys file.

services:
  sshtunnel:
    image: linuxserver/openssh-server:latest
    ...
    ports:
      - 2222:2222
    networks:
      - tunnel

networks:
  tunnel:
    external: true

Why can I only connect via tunnel after once connecting via Adminer?