I have a golang test that setup multiple stand alone docker containers using dockertest lib, and a k8s cluster that I setup using k3s.
As part of the test, I’m creating a docker network with name “my-docker-network”. I need the docker containers to be able to access the k8s cluster, how can I do that?
For example:
func setup(ctx context.Context, pool *dockertest.Pool, network *dockertest.Network) error {
resource, err := pool.RunWithOptions(&dockertest.RunOptions{
Name: myContainerName,
Repository: "my-container-repo",
Tag: "my-tag",
Networks: []*dockertest.Network{network},
})
if err != nil {
// ....
}
k3sContainer, err := k3s.Run(ctx, "docker.io/rancher/k3s:v1.27.1-k3s1",
network.WithNetwork([]string{}, &testcontainers.DockerNetwork{Name: network.Network.Name}),
testcontainers.WithConfigModifier(func(config *container.Config) {
config.Cmd = []string{
"server",
"--disable=traefik",
"--tls-san=host.docker.internal",
}
}))
if err != nil {
// ....
}
// ...
}
In the docker container, I have this code:
decodedCert, err := b64.StdEncoding.DecodeString(clusterCa)
if err != nil {
log.Fatal(err)
}
k8sConf := &rest.Config{
Host: clusterEndpoint,
BearerToken: accessToken,
TLSClientConfig: rest.TLSClientConfig{CAData: decodedCert},
}
k8sClient, err := kubernetes.NewForConfig(k8sConf)
if err != nil {
log.Fatal(err)
}
_, err = k8sClient.Discovery().RESTClient().Get().AbsPath("/healthz").DoRaw(context.Background())
if err != nil {
log.Fatal("Failed to get k8s /healthz", err)
}
Which return error:
Failed to get k8s /healthzGet "https://localhost:62034/healthz": dial tcp connect: connection refused
Everything work as expected when I try this code in localhost, so it’s not issue with the certificate or token, I’m also able to make it work by calling https://host.docker.internal:62034/healthz, however it’s not working in a github runner (“dial tcp: lookup host.docker.internal on 127.0.0.150:53: no such host”).
I wonder how I can setup k3s in a way that I will be able to call it from other docker containers in the same network using https://my-k3s-container:{EXPOSED_PORT}
, like I’m doing with the docker containers- I’m able to comminucate between two containers by using the conatiner name I’m passing in dockertest.RunOptions