The first time we deployed Celery to production on a client project, we thought we had done everything right. We had workers running, tasks queuing, and Redis as the broker. Six weeks later, the task queue was backed up with 40,000 unprocessed jobs, the workers had silently died, nobody knew, and a batch of client invoices had not been generated for two weeks.
That was four years ago. Since then we have deployed Celery on dozens of projects and we have learned what actually goes wrong — not in development, where everything works, but in production, where things fail in ways you do not anticipate.
This post covers the configuration and operational patterns we now use on every Celery deployment.
Why tasks fail silently (and how to stop it)
The most dangerous thing about Celery is how quietly it can fail. A worker process dies, the task queue fills up, and your application keeps accepting work and sending it to a queue that nobody is processing. No exception is raised. No alert fires. Users notice eventually, or you notice when a daily report does not arrive.
The fix has two parts: monitoring and task acknowledgement configuration.
Task acknowledgement
By default, Celery acknowledges a task (removes it from the queue) as soon as a worker picks it up, before the task runs. If the worker dies mid-task, the task is lost.
# celery.py
app = Celery('myproject')
app.conf.update(
# Only acknowledge after the task completes successfully
task_acks_late=True,
# If a worker dies, reject the task back to the queue
task_reject_on_worker_lost=True,
# Limit memory — workers that leak memory will restart cleanly
worker_max_memory_per_child=200_000, # 200MB in KB
# Limit tasks per child process to prevent long-running workers
# from accumulating state
worker_max_tasks_per_child=1000,
)
With task_acks_late=True, a task that is picked up by a dying worker will be requeued and picked up by another worker. The task might run twice (more on that shortly), but it will not be silently dropped.
Monitoring with Flower
Flower is a real-time web UI for Celery. We deploy it on every project:
pip install flower
celery -A myproject flower --port=5555
More usefully, configure it to report metrics to a monitoring system:
# celery.py
app.conf.update(
# Enable events so Flower and monitoring tools can track tasks
worker_send_task_events=True,
task_send_sent_event=True,
)
We alert on two Flower metrics: queue depth (if a queue has more than 500 tasks, something is wrong) and worker count (if the number of active workers drops below the expected minimum, workers have died).
Retry strategy
Tasks fail. The database is momentarily unavailable, an external API returns a 503, a file is not there yet. The question is not whether tasks will fail but whether they will fail gracefully.
from celery import shared_task
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
@shared_task(
bind=True,
max_retries=5,
default_retry_delay=60, # seconds
)
def send_invoice(self, invoice_id: int):
try:
invoice = Invoice.objects.get(id=invoice_id)
result = email_client.send_invoice(invoice)
invoice.mark_sent(result.message_id)
except Invoice.DoesNotExist:
# Don't retry — the invoice is genuinely gone
logger.error(f"Invoice {invoice_id} not found, not retrying")
return
except EmailServiceUnavailable as exc:
# Retry with exponential backoff
raise self.retry(
exc=exc,
countdown=60 * (2 ** self.request.retries), # 60s, 120s, 240s...
)
except Exception as exc:
logger.exception(f"Unexpected error sending invoice {invoice_id}")
raise self.retry(exc=exc)
A few things in this pattern:
bind=True gives the task access to self, which you need for self.retry(). The retry raises the exception back to Celery, which handles the requeue. If you call self.retry() without raise, the task continues executing after the retry is scheduled — which is almost never what you want.
The countdown for EmailServiceUnavailable uses exponential backoff — each retry waits longer than the last. If an email service is down, hammering it with retries at 60-second intervals just makes things worse.
Idempotency — designing tasks to be safe to run twice
With task_acks_late=True, a task might run more than once if a worker dies at exactly the wrong moment. This is fine if your tasks are idempotent — if running a task twice produces the same result as running it once.
Not all tasks are naturally idempotent. Sending an email twice is not the same as sending it once. Creating a record twice creates a duplicate.
The pattern we use:
@shared_task(bind=True, max_retries=3)
def send_welcome_email(self, user_id: int):
from django.core.cache import cache
lock_key = f"welcome_email_sent:{user_id}"
# Prevent duplicate sends using cache as a lock
if cache.get(lock_key):
logger.info(f"Welcome email for user {user_id} already sent, skipping")
return
try:
user = User.objects.get(id=user_id)
email_client.send_welcome(user.email, user.first_name)
# Set the lock with a TTL longer than any retry window
cache.set(lock_key, True, timeout=86400) # 24 hours
except User.DoesNotExist:
return
except Exception as exc:
raise self.retry(exc=exc)
For tasks that create database records, use get_or_create or update_or_create rather than create:
@shared_task
def create_monthly_report(account_id: int, month: str):
report, created = MonthlyReport.objects.get_or_create(
account_id=account_id,
month=month,
defaults={'status': 'generating'},
)
if not created and report.status == 'complete':
return # Already done
# Generate and save report...
Separate queues for different task types
A common mistake is routing all tasks to a single default queue. When a long-running task (generating a 10,000-row CSV export) enters the queue alongside time-sensitive tasks (sending a password reset email), the password reset waits behind the export.
We define separate queues by task priority and duration:
# celery.py
app.conf.task_queues = {
'critical': {
'exchange': 'critical',
'routing_key': 'critical',
},
'default': {
'exchange': 'default',
'routing_key': 'default',
},
'slow': {
'exchange': 'slow',
'routing_key': 'slow',
},
}
app.conf.task_default_queue = 'default'
app.conf.task_routes = {
'myapp.tasks.send_password_reset': {'queue': 'critical'},
'myapp.tasks.send_welcome_email': {'queue': 'critical'},
'myapp.tasks.generate_export': {'queue': 'slow'},
'myapp.tasks.process_webhook': {'queue': 'default'},
}
Then start separate workers for each queue:
# More workers on critical, fewer on slow
celery -A myproject worker -Q critical -c 4
celery -A myproject worker -Q default -c 4
celery -A myproject worker -Q slow -c 2
A slow export job can now block the slow queue without affecting password resets at all.
Celery Beat for scheduled tasks
Celery Beat is the scheduler — it triggers periodic tasks (daily reports, nightly cleanup jobs, hourly syncs). The common mistake is running Beat alongside a worker in the same process. Beat should run as a separate, single process:
# Never do this in production:
# celery -A myproject worker --beat
# Do this instead:
celery -A myproject beat --loglevel=info --scheduler django_celery_beat.schedulers:DatabaseScheduler
celery -A myproject worker -Q default -c 4
django-celery-beat stores schedules in the database rather than a flat file, which means you can update schedules at runtime via the admin without restarting Beat:
# settings.py
INSTALLED_APPS = [
...
'django_celery_beat',
]
# Define schedules in code as a fallback
app.conf.beat_schedule = {
'generate-nightly-reports': {
'task': 'myapp.tasks.generate_nightly_reports',
'schedule': crontab(hour=2, minute=0), # 2am daily
},
'cleanup-expired-sessions': {
'task': 'myapp.tasks.cleanup_expired_sessions',
'schedule': crontab(hour=3, minute=0), # 3am daily
},
}
Supervisor and process management
In production, workers need to restart automatically if they crash. We use Supervisor:
; /etc/supervisor/conf.d/celery.conf
[program:celery-worker-default]
command=/path/to/venv/bin/celery -A myproject worker -Q default -c 4 --loglevel=info
directory=/path/to/project
user=appuser
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
killasgroup=true
priority=998
stdout_logfile=/var/log/celery/worker-default.log
stderr_logfile=/var/log/celery/worker-default.log
[program:celery-beat]
command=/path/to/venv/bin/celery -A myproject beat --loglevel=info --scheduler django_celery_beat.schedulers:DatabaseScheduler
directory=/path/to/project
user=appuser
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=60
killasgroup=true
priority=999
stdout_logfile=/var/log/celery/beat.log
stderr_logfile=/var/log/celery/beat.log
stopwaitsecs=600 gives workers up to 10 minutes to finish their current task before a restart kills them. Without this, a deployment restart can kill tasks mid-execution.
The honest summary
Celery is reliable when configured correctly. The defaults are not designed for production — they are designed to get you up and running quickly. The gap between “working in development” and “working reliably in production” is mostly about:
- Late acknowledgement so tasks survive worker deaths
- Retry logic with exponential backoff
- Idempotency so retries do not cause duplicate actions
- Separate queues so slow tasks cannot block fast ones
- Process management so workers restart automatically
- Monitoring so you know when something is wrong
None of this is complicated to set up. It just requires knowing that it needs to be set up — which is what most Celery tutorials skip.
Lycore builds production Django applications for businesses — backend systems, task pipelines, API integrations, and custom software. Get in touch if you’re building something that needs to run reliably.
