Problem
I am running MacOS Sonoma with a fresh install of Podman v5.2.0.
My problem started when I deployed a standard NodeJS pod on Debian and a NodeJS application to that pod. The pod tag is:
node:20-bookworm-slim
In this pod, my NodeJS app watches for filesystem changes and automatically reloads with
npm run build:css -- --watch
However, my app fails to start because I run out of file watchers. I get the error
Error: ENOSPC: System limit for number of file watchers reached
Fixes Attempted
Pod Fix Attempted
I tried all of the standard methods for increasing fs.inotify.max_user_watches such as
echo 'fs.inotify.max_user_watches=524288' | tee -a /etc/sysctl.conf && sysctl -p
that results in the error
sysctl: permission denied on key "fs.inotify.max_user_watches"
I also tried to edit the value directly with
echo 524288 | dd of=/proc/sys/fs/inotify/max_user_watches
but that results in the error
dd: failed to open '/proc/sys/fs/inotify/max_user_watches': Read-only file system
It is impossible to change these inotify values from inside of the pod, no matter what I try.
Host Fix Attempted
After reading that Docker & Podman inherit most of the sysctl values from the host, I tried manually setting my local MacOS maxfiles value with:
sudo sysctl -w kern.maxfiles=524288
But, when I relauched the Debian pod and checked it’s limit with:
cat /proc/sys/fs/inotify/max_user_watches
the result is
8192
So, changing this value on my Mac had no impact at all on the value in the pod.
Podman v5.2.0 does not launch pods using MacOS as the host. Instead, Podman launches new pods with it’s own virtual machine called the Podman Default Machine. The values that new pods inherit do not come from your system, but from that virtual machine.
In order to edit this value, you need to SSH into the Podman Default Machine. It is locked by default, so you need to enable access to that machine in:
Settings > Preferences > Extension: Podman > Remote
You want to make sure the “Load remote system connections (ssh)” is Enabled.
Then, from your terminal access the Podman Default Machine with
podman machine ssh
That should load you into the default CentOS container as root. From there, you can simply run
echo 'fs.inotify.max_user_watches=524288' | tee -a /etc/sysctl.conf && sysctl -p
to permanently update the max_user_watches parameter. This parameter is then inherited by any new pods that are launched.
NOTE: This should work for any sysctl work you need to do on your pods. You can not edit many of those values from the pod, so you need to edit them in the Podman Default Machine and inherit them on launch.
1