Im currently trying to make docker work inside a docker deployment, the reason is that my structure looks like this:

• meetingsd-headless

  This is a C++ SDK that can be called using docker compose up to execute, it also has a Dockerfile and docker-compose

• app.py

  Responsible for endpoint which when triggered would call subprocess docker compose up

• Dockerfile

  Using this to wrap and deploy to pipeline

• docker-compose.yml

I know it sounds inefficient to have docker inside a docker instance but it’s an approach I would like to try, while researching I found out the docker socket would help so this is my attempt:

docker-compose.yml

version: '3.8'

services:
  flask_app:
    build: .
    ports:
      - "5007:5007"
    volumes:
      - .:/app
      - /var/run/docker.sock:/var/run/docker.sock  # Share the host Docker socket
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json  # Adjust the path accordingly
      - DOCKER_HOST=unix:///var/run/docker.sock  # Set the Docker host to the Unix socket

Dockerfile:

# Use the official Python image from the Docker Hub
FROM python:3.9

# Install Docker CLI
RUN apt-get update && 
    apt-get install -y docker.io && 
    rm -rf /var/lib/apt/lists/*

# Set the working directory
WORKDIR /app

# Copy the requirements.txt file and install the dependencies
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

# Copy the rest of the application code
COPY . .

# Expose the port the app runs on
EXPOSE 5007

# Run the application
CMD ["python", "app_multi.py"]

Below are parts of my app.py code:

In this part I’m verifying if docker is installed, so far it seems not even with my Dockerfile and docker-compose

app = Flask(__name__)

# Debugging: Print Docker environment variable
print("DOCKER_HOST:", os.environ.get("DOCKER_HOST"))

# Debugging: Verify Docker client connection
try:
    client = docker.from_env()
    print("Docker client initialized successfully.")
except Exception as e:
    print(f"Error initializing Docker client: {e}")

And this part is how it calls docker compose to the sdk folder written in C++

# Start the Docker container
process = subprocess.Popen(
    ["docker", "compose", "-f", "/home/seveneleven/PycharmProjects/PropZoom/meetingsdk-headless/compose.yaml", "up"],
)
container_name = 'meetingsdk-headless-zoomsdk-1'  # The name of your Docker container
if wait_for_container_to_start(container_name):
    meetings[meeting_id]['status'] = 'running'
    thread = threading.Thread(target=monitor_docker_logs, args=(container_name, meeting_id))
    thread.start()
process.wait()

What can be done to make this hybrid code work? I need to dockerize the API because we use pipeline and Azure clusters

After a few suggestions in comments I tried out DIND (Docker-in-docker) and below is my new compose:

version: '3.8'

services:
  dind:
    image: docker:24.0.9-dind  # Use the latest Docker-in-Docker image
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
      - DOCKER_HOST=tcp://dind:2375  # Set the Docker host to the DinD service

    volumes:
      - dind-certs-ca:/certs/ca
      - dind-certs-client:/certs/client
      - /var/run/docker.sock:/var/run/docker.sock  # Share the Docker socket with the host

  flask_app:
    build: .
    ports:
      - "5007:5007"
    volumes:
      - .:/app
    depends_on:
      - dind
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json  # Adjust the path accordingly
      - DOCKER_HOST=tcp://dind:2375  # Set the Docker host to the DinD service
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1
    volumes:
      - dind-certs-client:/certs/client


volumes:
  dind-certs-ca:
  dind-certs-client:

I’m still getting that docker client is not initialized properly via this code, I even tried forcing the tcp string in the base_url:

try:
    client = docker.DockerClient(base_url=os.environ.get("DOCKER_HOST"))
    print("Docker client initialized successfully.")
except Exception as e:
    print(f"Error initializing Docker client: {e}")
    client = None