Symfony version(s) affected
7.0
We are using symfony messenger directly through kubernates command, without supervisor, example of configuration:
command:
enabled: true
cli: ‘[“/bin/sh”]’
args: ‘[“-c”, “php bin/console messenger:consume default –time-limit=3600 –memory-limit=256M”]’
We expecting that after reaching 3600 seconds since starting messenger will finish work with exit code “0” and pod will gracefully reload worker – and in most cases it happens as it should, but sometimes worker freezes out without any exit code
The latest what we see in STDOUT is:
{“message”:”Worker stopped due to time limit of 3600s exceeded”,”context”:{},”level”:200,”level_name”:”INFO”,”channel”:”messenger”,”datetime”:”2024-04-10T08:39:37.697238+00:00″,”extra”:{}}
after that time of latest logs will stay as it forever. Meanwhile kubernates still treating pod with this idle state as normally working.
Issue normally happened rare, that’s why it is hard to provide exact steps to reproduce issue. To reproduce need to run pod with provided configuration and wait when worker will be go to this “idle” state. It can take few hours or even days, issue have sporadical nature
We have two possible strategy to resolve that issue:
- Use messenger with supervisord
- Handle WorkerStoppedEvent and through an exception to force pod recreating
And both solutions not ideal
If anyone can help with this tricky case it will be very helpfull