For the past few days, my Docker containers have been generating connection errors between them. For example, I have a container that sends HTTP requests to another container that publishes port 8008 to send emails. The second container receives the request, connects to a mail server, and sends the email. I’ve checked everything and can’t find the problem. Here is some more context
these are my containers:
[user@host ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
68250db6e6d6 amnd_api:1.1 "gunicorn -w 20 -b 0…" 10 months ago Up 16 hours 8004/tcp amnd_api
0ca9c384108e amnd_proxy:1.0 "/docker-entrypoint.…" 11 months ago Up 16 hours 80/tcp, 10.161.79.6:443->443/tcp amnd_proxy
c4e229bd6de0 amnd_monitor:1.0 "/usr/bin/supervisor…" 11 months ago Up 16 hours amnd_monitor
9b4a94db6b37 amnd_email:1.0 "/usr/bin/supervisor…" 11 months ago Up 16 hours 8008/tcp amnd_email
baefb4811314 postgres:14.2 "docker-entrypoint.s…" 11 months ago Up 16 hours 5432/tcp amnd_database
and this is the network they are on:
[user@host ~]$ docker network inspect amnd_microservices
[
{
"Name": "amnd_microservices",
"Id": "f1a220b04c716407c54a944144cb85d09ee4c10677983aa2bc2c69df7bddc1c1",
"Created": "2023-05-17T13:03:48.493942255-05:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "192.168.115.0/24",
"Gateway": "192.168.115.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"0ca9c384108ec95dbd9b4ef3e644fdc0f3571247bed6c8f48eb5869a1e0d4f1b": {
"Name": "amnd_proxy",
"EndpointID": "a7630fc905059e7df748c08b0bd07ebad094327725651cc6e5f3a7c42018cb1d",
"MacAddress": "02:42:c0:a8:73:03",
"IPv4Address": "192.168.115.3/24",
"IPv6Address": ""
},
"68250db6e6d631374af418d34e36ec50e7bb0824ef7a4b6c9855f4f02b309bad": {
"Name": "amnd_api",
"EndpointID": "6e101b7ef3e275ea005bb7bba3c0dd13ff72f239448537816b961bf8214fe553",
"MacAddress": "02:42:c0:a8:73:04",
"IPv4Address": "192.168.115.4/24",
"IPv6Address": ""
},
"9b4a94db6b37971cc1670ad540e2137caf4f74af032fbca8c86d6711f6d5b5e3": {
"Name": "amnd_email",
"EndpointID": "f2056993ebb15298846a221178decf82a4a53743984da95f77b910e0dcaaf26a",
"MacAddress": "02:42:c0:a8:73:08",
"IPv4Address": "192.168.115.8/24",
"IPv6Address": ""
},
"baefb4811314ada3db4bacba13d58ec247cdeeebcc5a3c2b89a908b903c42491": {
"Name": "amnd_database",
"EndpointID": "1b73d3701caf5ab550e84645233e546c58a171f7bbd4f46a54aa4dc1abfdb03e",
"MacAddress": "02:42:c0:a8:73:02",
"IPv4Address": "192.168.115.2/24",
"IPv6Address": ""
},
"c4e229bd6de074cd5b26853d6572d025bf54f99e580384f5ac92c5ba5a1242d8": {
"Name": "amnd_monitor",
"EndpointID": "babfcd878476a74bfdec2e8c7af8895dcdf8cf4fd619d1a53b73c0f39d6d6ad8",
"MacAddress": "02:42:c0:a8:73:07",
"IPv4Address": "192.168.115.7/24",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "microservices",
"com.docker.compose.project": "amnd",
"com.docker.compose.version": "2.17.3"
}
}
]
I’m testing the connection between amnd_monitor and amnd_email. When I ping amnd_email, it responds without a problem, but when I try to connect to it via HTTP, it doesn’t respond.
This is a tcpdump from the amnd_email container showing that the ping arrives and is responded to. There’s something I don’t understand. Why is the source address 192.168.115.1, which is the gateway of the container network? That’s what the tcpdump shows me, amnd_monitor is 192.168.115.7:
9b4a94db6b37:/app# tcpdump -i eth0 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:48:52.117336 IP 192.168.115.1 > 192.168.115.8: ICMP echo request, id 9042, seq 1, length 64
08:48:52.117359 IP 192.168.115.8 > 192.168.115.1: ICMP echo reply, id 9042, seq 1, length 64
08:48:53.173359 IP 192.168.115.1 > 192.168.115.8: ICMP echo request, id 9042, seq 2, length 64
08:48:53.173385 IP 192.168.115.8 > 192.168.115.1: ICMP echo reply, id 9042, seq 2, length 64
Moving on, when I make HTTP requests from amnd_monitor, they are not visible in the amnd_email tcpdump. However, I see the following in the host tcpdump:
[user@host ~]$ sudo tcpdump -i br-f1a220b04c71 port 8008
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-f1a220b04c71, link-type EN10MB (Ethernet), capture size 262144 bytes
09:00:15.638670 IP 192.168.115.7.53865 > 192.168.115.8.8008: Flags [S], seq 1884919415, win 0, length 0
09:00:15.640335 IP 192.168.115.7.53865 > 192.168.115.8.8008: Flags [S], seq 1884919415, win 0, length 0
09:00:15.651360 IP 192.168.115.7.53865 > 192.168.115.8.8008: Flags [S], seq 1884919415, win 0, length 0
Finally, if I make the requests from the host machine, there is a response from amnd_email and I see the packets in the tcpdump:
9b4a94db6b37:/app# tcpdump -i eth0 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:51:06.611815 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [S], seq 533359903, win 64240, options [mss 1460,sackOK,TS val 2173420530 ecr 0,nop,wscale 7], length 0
08:51:06.611856 IP 192.168.115.8.8008 > 192.168.115.1.39096: Flags [S.], seq 952322982, ack 533359904, win 65160, options [mss 1460,sackOK,TS val 3023416566 ecr 2173420530,nop,wscale 7], length 0
08:51:06.611904 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [.], ack 1, win 502, options [nop,nop,TS val 2173420530 ecr 3023416566], length 0
08:51:06.612105 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [P.], seq 1:92, ack 1, win 502, options [nop,nop,TS val 2173420530 ecr 3023416566], length 91
08:51:06.612116 IP 192.168.115.8.8008 > 192.168.115.1.39096: Flags [.], ack 92, win 509, options [nop,nop,TS val 3023416566 ecr 2173420530], length 0
08:51:06.634625 IP 192.168.115.8.8008 > 192.168.115.1.39096: Flags [P.], seq 1:193, ack 92, win 509, options [nop,nop,TS val 3023416588 ecr 2173420530], length 192
08:51:06.634712 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [.], ack 193, win 501, options [nop,nop,TS val 2173420553 ecr 3023416588], length 0
08:51:06.634752 IP 192.168.115.8.8008 > 192.168.115.1.39096: Flags [P.], seq 193:346, ack 92, win 509, options [nop,nop,TS val 3023416589 ecr 2173420553], length 153
08:51:06.634777 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [.], ack 346, win 501, options [nop,nop,TS val 2173420553 ecr 3023416589], length 0
08:51:06.634899 IP 192.168.115.8.8008 > 192.168.115.1.39096: Flags [F.], seq 346, ack 92, win 509, options [nop,nop,TS val 3023416589 ecr 2173420553], length 0
08:51:06.634917 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [F.], seq 92, ack 346, win 501, options [nop,nop,TS val 2173420553 ecr 3023416589], length 0
08:51:06.634939 IP 192.168.115.8.8008 > 192.168.115.1.39096: Flags [.], ack 93, win 509, options [nop,nop,TS val 3023416589 ecr 2173420553], length 0
08:51:06.634976 IP 192.168.115.1.39096 > 192.168.115.8.8008: Flags [.], ack 347, win 501, options [nop,nop,TS val 2173420553 ecr 3023416589], length 0
Has anyone else experienced this? Or any help diagnosing what might be happening? My applications are down because there is no connection between the containers even though they are on the same network. I only get a ping response between them.
I restarted the containers, but the problem persists. I’ve been running tcpdump captures and tests, and I suspect that there’s something in the firewall blocking those connections, but I haven’t been able to pinpoint the exact issue. The applications had been working flawlessly for about a year, and then this issue suddenly started. Checking the container logs, I see that the failure started after I set a route using nmtui and restarted the NetworkManager process. I’m not sure if this is related, but it’s worth checking.