I configures in my Ubuntu and Apache server a Django project in which I use celery to manage and handle backgound jobs/tasks.
The problem that I see is that when there is a queue of tasks and is managing 3, if the 4th, 5th… do not start executing within an hour they duplicate.
Here is what i have…
Supervisor conf
[program:project_app]
command=/path/celery -A haus worker --loglevel=INFO -c 3
directory=/path/
user=www-data
autostart=true
autorestart=true
stdout_logfile=/path/celeryd.log
redirect_stderr=true
celery.py
from future import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘project.settings’)
app = Celery(‘project’)
app.config_from_object(‘django.conf:settings’)
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
init.py
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)
settings.py
...
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'America/Mexico_City'
...
The way i execute is:
@shared_task()
def do_some(args):
# code
do_some.delay(args)
[2024-06-09 22:49:46,855: INFO/MainProcess] Task haus.celery_tasks.do_some[8bc9dc9e-493a-4b1d-b81a-f758c8ef49a3] received
[2024-06-09 22:49:57,021: INFO/MainProcess] Task haus.celery_tasks.do_some[a58ba0cb-4881-430d-b652-a8d4edd704ea] received
[2024-06-09 22:50:13,631: INFO/MainProcess] Task haus.celery_tasks.do_some[5cd7f9d3-c77f-4331-8459-9889e19f34af] received
[2024-06-09 22:52:25,640: INFO/MainProcess] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] received
[2024-06-09 22:52:51,268: INFO/MainProcess] Task haus.celery_tasks.do_some[54a35f23-426a-463f-9e72-e733cdd79ed2] received
[2024-06-09 22:52:58,362: INFO/MainProcess] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] received
[2024-06-09 23:53:43,860: INFO/MainProcess] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] received
[2024-06-09 23:53:43,898: INFO/MainProcess] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] received
[2024-06-09 23:53:43,909: INFO/MainProcess] Task haus.celery_tasks.do_some[54a35f23-426a-463f-9e72-e733cdd79ed2] received
[2024-06-10 00:32:37,245: INFO/ForkPoolWorker-3] Task haus.celery_tasks.do_some[8bc9dc9e-493a-4b1d-b81a-f758c8ef49a3] succeeded in 6170.361236825585s: None
[2024-06-10 00:32:51,723: INFO/ForkPoolWorker-2] Task haus.celery_tasks.do_some[a58ba0cb-4881-430d-b652-a8d4edd704ea] succeeded in 6174.678496623412s: None
[2024-06-10 00:32:58,670: INFO/ForkPoolWorker-4] Task haus.celery_tasks.do_some[5cd7f9d3-c77f-4331-8459-9889e19f34af] succeeded in 6165.020594196394s: None
[2024-06-10 02:11:18,380: INFO/ForkPoolWorker-1] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] succeeded in 6056.458753755316s: None
[2024-06-10 02:15:05,588: INFO/ForkPoolWorker-3] Task haus.celery_tasks.do_some[54a35f23-426a-463f-9e72-e733cdd79ed2] succeeded in 6148.0511863250285s: None
[2024-06-10 02:15:50,487: INFO/ForkPoolWorker-2] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] succeeded in 6178.357154054567s: None
[2024-06-10 02:15:55,220: INFO/ForkPoolWorker-4] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] succeeded in 6176.377053318545s: None
[2024-06-10 03:52:21,739: INFO/ForkPoolWorker-1] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] succeeded in 6063.125598752871s: None
[2024-06-10 03:56:08,022: INFO/ForkPoolWorker-3] Task haus.celery_tasks.do_some[54a35f23-426a-463f-9e72-e733cdd79ed2] succeeded in 6062.099442152306s: None
Here you can see how the first 2 are executed once and the last 2 repeated or reexecuted
[2024-06-09 22:49:46,855: INFO/MainProcess] Task haus.celery_tasks.do_some[8bc9dc9e-493a-4b1d-b81a-f758c8ef49a3] received
[2024-06-09 22:49:57,021: INFO/MainProcess] Task haus.celery_tasks.do_some[a58ba0cb-4881-430d-b652-a8d4edd704ea] received
[2024-06-09 22:50:13,631: INFO/MainProcess] Task haus.celery_tasks.do_some[5cd7f9d3-c77f-4331-8459-9889e19f34af] received
[2024-06-09 22:52:25,640: INFO/MainProcess] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] received
[2024-06-09 22:52:58,362: INFO/MainProcess] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] received
[2024-06-09 23:53:43,860: INFO/MainProcess] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] received
[2024-06-09 23:53:43,898: INFO/MainProcess] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] received
[2024-06-10 00:32:37,245: INFO/ForkPoolWorker-3] Task haus.celery_tasks.do_some[8bc9dc9e-493a-4b1d-b81a-f758c8ef49a3] succeeded in 6170.361236825585s: None
[2024-06-10 00:32:51,723: INFO/ForkPoolWorker-2] Task haus.celery_tasks.do_some[a58ba0cb-4881-430d-b652-a8d4edd704ea] succeeded in 6174.678496623412s: None
[2024-06-10 00:32:58,670: INFO/ForkPoolWorker-4] Task haus.celery_tasks.do_some[5cd7f9d3-c77f-4331-8459-9889e19f34af] succeeded in 6165.020594196394s: None
[2024-06-10 02:11:18,380: INFO/ForkPoolWorker-1] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] succeeded in 6056.458753755316s: None
[2024-06-10 02:15:50,487: INFO/ForkPoolWorker-2] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] succeeded in 6178.357154054567s: None
[2024-06-10 02:15:55,220: INFO/ForkPoolWorker-4] Task haus.celery_tasks.do_some[b6190664-179e-4316-8275-3b184e5b3cdc] succeeded in 6176.377053318545s: None
[2024-06-10 03:52:21,739: INFO/ForkPoolWorker-1] Task haus.celery_tasks.do_some[cd98b3e3-022d-4785-a89e-128a659c8e95] succeeded in 6063.125598752871s: None
Help please
I need each one to be executed only once
Isaac Harari is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.