I have an application running every 5 minutes which utilizes Docker and PBS for its execution.
The application is a python script which is executed via PBS script in a docker container.
This is the PBS script which uses rm_test docker image for executing _regmod_realtime_forecast.py script
The architecture works fine most of the time but, Once in a blue-moon, the PBS comes across an orphan container. In this situation, The pbs terminates prematurely by partially executing the underlying python script.
PBS logs of an irregular execution
PBS logs of a healthy execution
If we observe closely, the resource used in both the scenarios imply the abnormal behavior when the PBS comes across an orphan docker container.

I am using the same image for performing other jobs as well.
May be they are creating issue and storing orphan containers.
So i was thinking of having a separate image dedicated for execution of my application.