Issue with Podman Rootless Setup for Nginx Proxy Manager

I’ve been trying to migrate from Docker on my old home server to a rootless Podman setup on a new server. The setup works perfectly on my laptop but fails on the new server. Below are the details of my setup and the error I’m encountering. Any help would be greatly appreciated.

docker-compose.yml:

services:
  nginx-proxy-manager:
    image: 'docker.io/lepresidente/nginx-proxy-manager:latest'
    ports:
      - '80:80'
      - '443:443'
      - '81:81'
    environment:
      DB_MYSQL_HOST: ${DB_MYSQL_HOST}
      DB_MYSQL_PORT: ${DB_MYSQL_PORT}
      DB_MYSQL_USER: ${DB_MYSQL_USER}
      DB_MYSQL_PASSWORD: ${DB_MYSQL_PASSWORD}
      DB_MYSQL_NAME: ${DB_MYSQL_NAME}
    env_file:
      - .env
    depends_on:
      - mariadb
    volumes:
      - data:/data:z
      - ssl:/etc/letsencrypt/:z
      - npm_config:/config:z
    restart: unless-stopped

  mariadb:
    image: lscr.io/linuxserver/mariadb:latest
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_MYSQL_NAME}
      MYSQL_USER: ${DB_MYSQL_USER}
      MYSQL_PASSWORD: ${DB_MYSQL_PASSWORD}
    env_file:
      - .env
    volumes:
      - db_config:/config
      - db:/var/lib/mysql
    restart: unless-stopped

volumes:
  data:
  ssl:
  db_config:
  npm_config:
  db:

.env:

TZ=Europe/Berlin
GUID=1000
PGID=1000

# npm
DB_MYSQL_HOST=mariadb
DB_MYSQL_PORT=3306
DB_MYSQL_USER=npm_user
DB_MYSQL_PASSWORD=XXXXXX
DB_MYSQL_NAME=nginx_proxy_manager

# mariadb
MYSQL_ROOT_PASSWORD=XXXXX

podman info:

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.47
    systemPercent: 0.3
    userPercent: 0.24
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2041
  hostname: heimserver
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-39-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 13308612608
  memTotal: 15639355392
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  store:
    configFile: /home/lettner/.config/containers/storage.conf
    containerStore:
      number: 2
      paused: 0
      running: 1
      stopped: 1
    graphDriverName: overlay
    graphRoot: /home/lettner/.local/share/containers/storage
    graphRootAllocated: 105089261568
    graphRootUsed: 10203324416
    graphStatus:
      Backing Filesystem: extfs
      Native Overlay Diff: "true"
      Supports d_type: "true"
      Supports shifting: "false"
      Supports volatile: "true"
      Using metacopy: "false"
    imageCopyTmpDir: /var/tmp
    imageStore:
      number: 3
    runRoot: /run/user/1000/containers
    transientStore: false
    volumePath: /home/lettner/.local/share/containers/storage/volumes
  version:
    APIVersion: 4.9.3
    Built: 0
    BuiltTime: Thu Jan  1 00:00:00 1970
    GitCommit: ""
    GoVersion: go1.22.2
    Os: linux
    OsArch: linux/amd64
    Version: 4.9.3

My Setup

• Server OS: Ubuntu 24.04 LTS x86_64
• Podman Version: 4.9.3
• OCI Runtime: crun
• Kernel: 6.8.0-39-generic

The Issue

When trying to start the containers with podman-compose, I encounter the following error:

Error: crun: creating `/etc/letsencrypt/`: openat2 `etc/letsencrypt`: No such file or directory: OCI runtime attempted to invoke a command that was not found
exit code: 127
podman start nginx-proxy-manager_nginx-proxy-manager_1
Error: unable to start container "a7f05523b12a2590fbecc007f8a43b8899fcb564925ce5e9954e534a1406c9b1": crun: creating `/etc/letsencrypt/`: openat2 `etc/letsencrypt`: No such file or directory: OCI runtime attempted to invoke a command that was not found
exit code: 125

What I Tried

1. Filesystem Permissions:

   • Ensured the Podman user has access to the directories.
   • Verified and adjusted ownership and permissions of the directories.

2. AppArmor:

   • Temporarily disabled AppArmor to check if it was causing the issue.

3. SELinux:

   • Set SELinux to permissive mode (though it’s disabled in podman info).

4. Volume Mounting:

   • Ensured the volumes are correctly created and inspected them.

Comparison with Laptop (Working Setup)

• Laptop OS: Arch Linux x86_64
• Kernel: 6.10.1-arch1-1
• Environment: GNOME 46.3.1

Questions

1. Are there specific SELinux or AppArmor configurations I need to adjust for Podman?
2. Are there any differences in Podman setup between Arch Linux and Ubuntu that could cause this issue?
3. Any other suggestions for resolving the permission issue?

Thanks in advance for any help or suggestions!