We have a Debian machine here on which we are running multiple Docker containers used to run Python scripts. Some of these scripts use a significant proportion of our machine’s memory. It even happens from time to time that a process uses too much and crashes due to OOM (and we have to rework the script to somehow make it use less memory).
This is due to our scripts processing significant data volumes.
I would like to be able to assign memory priorities between running containers. Essentially be able to pause running containers (except for the one with highest priority) if memory becomes scarce. I know I could assign soft memory limits but then the containers would stop when their limit is reached, making the python script running inside crash. Is there a smart way I could automatically pause a container (and not stop it) if it reaches a certain memory usage? And have it start again when memory becomes available?
We tried using the memory limits of Docker containers, but then the Python script running inside the container crashes when the container is stopped so this is a bit problematic for us.
36821 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
You can manipulate RAM priority in some way with mlock. Basically you can prevent one process memory to go to swap (where performance would be much worse). The others would go to swap, giving you some form of priority system.
More discussion on mlock here
You can also try to prioritise processes on a Linux system. It might help indirectly.
I’m assuming each of these containers have its own process. I’m not sure if you’re on a VPS or a personal machine, but this shouldn’t matter.
In that case, you can use the nice command. You might have to restart your containers so you can give them custom priority.
If your process is already running, you can user renice to just change priority.
nice -n -10 container_process
Lower values have higher priorities. Highest typically goes to -20.
Similar issue
More examples
2