Celery Integration

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

This page is written against Django 6.1 (6.1.1, released 2026-09-02; 6.1.2 is scheduled for 2026-10-06), Python 3.12-3.14, Celery 5.6.3 (2026-03-26) and Flower 2.2.0 (2026-09-22), all checked on 2026-10-03; where the 5.2 LTS (5.2.17) behaves differently the text says so. Every example was run in a scratch project on Python 3.13 with Django 6.1.1 and SQLite, using Celery’s in-memory broker (memory://), an in-process result backend, the Django test runner, an embedded Celery worker and a real celery beat process. Docker, Redis, RabbitMQ and PostgreSQL were not available, so the Redis and RabbitMQ settings shown below were not run against a live broker; the text marks each place where that matters. Celery is the long-established way to run work outside the request/response cycle: sending the review-notification emails of the Bookshelf application, regenerating PDFs, calling slow third-party APIs, importing files, and running scheduled jobs. This page shows how to wire it into a Django project, how to write tasks that survive retries and duplicate delivery, how to schedule and monitor them, how to test them, and how to decide between Celery and the Tasks framework that ships with Django since 6.0 (see Tasks Framework).

In a hurry: what goes where

Need Rule of thumb Section

Connect Celery to Django

config/celery.py creates the app, reads CELERY_* settings, calls autodiscover_tasks(); config/__init__.py imports it.

Setting up Celery in a Django project

Write a task

@shared_task in catalog/tasks.py; arguments are JSON, so pass primary keys, never model instances.

Writing tasks with @shared_task

Call a task from a view

task.delay_on_commit(pk) inside the transaction; delay() only when nothing depends on the commit.

Calling tasks

Pick a broker

RabbitMQ or Redis for production, memory:// only for experiments; the broker is not the result store.

Brokers and result backends

Survive failures

autoretry_for plus retry_backoff, a max_retries, and a task body that is safe to run twice.

Retries and Idempotency and delivery guarantees

Run scheduled jobs

CELERY_BEAT_SCHEDULE and exactly one celery beat process, or django-celery-beat to edit the schedule in the admin.

Periodic tasks with celery beat

See what the workers are doing

Flower, behind authentication, with worker events enabled.

Monitoring with Flower

Test

Mock or call the function in unit tests; eager mode is a convenience, not a faithful emulation.

Testing Celery code

Decide whether you need it

Celery for retries, scheduling, routing and workflows today; django.tasks when a simple queue is enough.

Celery or django.tasks?

How Celery fits into a Django project

Celery is a separate distributed system that Django code talks to. It has four moving parts, and none of them runs inside the web process:

  • the producer is your Django code (a view, a signal receiver, a management command), which serializes a task call into a message;

  • the broker is a message transport (RabbitMQ or Redis in practice) that holds the messages until a worker takes one;

  • the worker is a long-running celery worker process that imports your Django project, so it has the ORM, settings and installed apps, and executes the task function;

  • the result backend is an optional store (Redis, a database, the Django ORM or cache) where the worker puts the state and return value so the caller can read them later.

celery beat is a fifth, optional process: a clock that publishes messages for periodic tasks to the same broker.

flowchart LR V["Django view"] -->|"delay_on_commit(review_id)"| B[("Broker: RabbitMQ or Redis")] BT["celery beat"] -->|"periodic messages"| B B --> W["celery worker (imports Django)"] W -->|"ORM reads and writes"| D[("Database")] W -->|"state and result"| R[("Result backend")] V -.->|"AsyncResult.get() (avoid in views)"| R

The request/response cycle (see Request and Response Cycle) only pays the cost of publishing a message; the slow work happens afterwards in another process. That is the whole value, and it also explains the two big sources of bugs: the worker runs later and elsewhere, so it may see different data than the view saw, and a message may be delivered more than once.

Source: Introduction to Celery and Backends and brokers

Setting up Celery in a Django project

Install Celery in the same environment as Django. The redis extra pulls the Redis client for a Redis broker or backend; RabbitMQ needs no extra package because AMQP support is built in.

Installing Celery
python -m pip install "celery[redis]"

Source: Using Redis

The documented layout for a modern project (manage.py next to a project package, here config) needs two files. The first creates the Celery application.

The Celery app (config/celery.py)
import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")

app = Celery("config")

# Read every setting that starts with CELERY_ from Django's settings.
app.config_from_object("django.conf:settings", namespace="CELERY")

# Look for a tasks.py module in each installed app.
app.autodiscover_tasks()

The second makes sure that the app is created whenever Django starts, so that @shared_task has an app to bind to.

Loading the app with Django (config/__init__.py)
# Make sure the Celery app is loaded when Django starts, so that
# @shared_task binds to it.
from .celery import app as celery_app

__all__ = ("celery_app",)

Source: First steps with Django

Three details deserve a sentence each:

  • config_from_object("django.conf:settings", namespace="CELERY") passes the settings by name, so the worker does not have to serialize the settings object to its child processes. The CELERY namespace means every Celery option is the upper-case form of its documented name with a CELERY_ prefix: task_always_eager becomes CELERY_TASK_ALWAYS_EAGER, broker_url becomes CELERY_BROKER_URL and worker_concurrency becomes CELERY_WORKER_CONCURRENCY. The prefix keeps Celery keys from colliding with other Django settings.

  • autodiscover_tasks() with no argument walks INSTALLED_APPS and imports tasks.py from each app that has one, so nobody has to maintain a CELERY_IMPORTS list.

  • The os.environ.setdefault line is optional but saves passing --settings to the celery command, and it must run before the app is created.

The settings go in the usual settings.py. Read anything environment-specific (above all the broker URL, which usually embeds credentials) from the environment, exactly as for SECRET_KEY.

Celery settings (config/settings.py)
import os

CELERY_BROKER_URL = os.environ["CELERY_BROKER_URL"]          # redis://… or amqp://… (set by the platform)
CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND")  # optional
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 5 * 60        # hard limit, seconds
CELERY_TASK_SOFT_TIME_LIMIT = 4 * 60   # raises SoftTimeLimitExceeded inside the task
CELERY_WORKER_PREFETCH_MULTIPLIER = 1  # fair scheduling for long tasks

Source: First steps with Django and Configuration and defaults

In the scratch project the same settings used os.environ.get("CELERY_BROKER_URL", "memory://") so that manage.py check and the tests ran with no infrastructure; do not give a production setting a silent in-memory default, because every process would then publish into its own private queue. Start a worker with the celery command; it prints the configuration and the tasks it discovered, which is the quickest check that autodiscovery worked:

Starting a worker
celery -A config worker -l INFO
 -------------- celery@host v5.6.3 (recovery)
…
[tasks]
  . catalog.tasks.count_reviews
  . catalog.tasks.moderate_review
  . catalog.tasks.notify_author
  . catalog.tasks.remind_unmoderated
…
[… INFO/MainProcess] celery@host ready.

Source: Starting the worker process

Writing tasks with @shared_task

Reusable apps cannot import the project’s Celery app, so Celery provides @shared_task: it creates a task that binds to whichever app is current, which with the __init__ import above is the project app. Put tasks in tasks.py so that autodiscovery finds them.

Tasks of the catalog app (catalog/tasks.py)
from celery import shared_task
from django.core.mail import send_mail
from django.utils import timezone

from .models import Review


@shared_task
def count_reviews(book_id):
    return Review.objects.filter(book_id=book_id).count()


@shared_task(ignore_result=True)
def moderate_review(review_id):
    # Claim the row atomically: only the first execution sees updated == 1,
    # so a duplicate delivery of the same message does nothing.
    updated = Review.objects.filter(pk=review_id, moderated_at__isnull=True).update(
        moderated_at=timezone.now()
    )
    return bool(updated)


@shared_task
def remind_unmoderated():
    pending = Review.objects.filter(moderated_at__isnull=True).count()
    return f"{pending} reviews waiting for moderation"

Source: Using the @shared_task decorator and Tasks

The rules that make a task robust are all about the fact that the call is serialized and runs later:

  • Arguments are JSON. Celery’s default task_serializer and result_serializer are json (the default since Celery 4.0, before that pickle) and accept_content is json only. Pass integers, strings, lists and dicts. The encoder that Celery uses (kombu) also round-tripped Decimal, date and UUID values in the scratch project (kombu 5.6.2), but model instances, querysets and sets raise kombu.exceptions.EncodeError; sending only primitives is the habit that keeps messages small and readable by any consumer. Do not enable pickle to make a difficult argument work: a message that can unpickle is a message that can run code, and the broker is a network service.

  • Pass identifiers, not model instances. The Celery documentation uses a Django model as its example of what not to send: by the time the worker runs, someone may have changed the row, and the task would write the stale copy back. Pass the primary key and re-fetch inside the task, as notify_author does below. Small messages are also cheaper.

  • The task is responsible for checking the world. If a job must not run more often than every five minutes, the task checks that, not the caller.

  • Keep the body thin. Put the business logic in a model method or a service function so that the same code can run from a view, a management command or a task, and test it without Celery (see Testing Celery code).

  • Opt out of results you do not read. ignore_result=True (per task) or CELERY_TASK_IGNORE_RESULT (global) stops the worker from writing a result nobody will fetch.

  • Name collisions are silent. The default task name is the dotted module path, catalog.tasks.count_reviews; keep tasks.py modules importable under one path, and note that the name, not the import path, is what CELERY_BEAT_SCHEDULE and routes use.

The same bind=True option that gives access to the task instance is what the retry examples use; self.request carries the task id, the retry count and the delivery information.

Calling tasks

delay(*args, **kwargs) is the shortcut for apply_async(args, kwargs); apply_async adds the execution options: countdown (seconds), eta (an aware datetime), expires, queue, priority, retry and more. Both return an AsyncResult immediately.

Calling a task with options
from catalog.tasks import count_reviews

count_reviews.delay(book.pk)
count_reviews.apply_async((book.pk,), countdown=30, expires=600, queue="default")

Source: Calling tasks

Enqueue after the transaction commits

The most common Django-specific bug is publishing the message before the database transaction that created the row has committed. The worker is fast, the commit is not, and Review.objects.get(pk=review_id) raises DoesNotExist in the worker. The fix is Django’s transaction.on_commit() (see Transactions, Raw SQL and Multiple Databases), and since Celery 5.4 the task object has a shortcut for it, delay_on_commit(), available on every task created through the documented setup because Celery then uses its DjangoTask base class.

Enqueueing from a view after commit (catalog/views.py)
from django.db import transaction
from django.http import JsonResponse
from django.shortcuts import get_object_or_404
from django.views.decorators.http import require_POST

from .models import Book, Review
from .tasks import moderate_review, notify_author


@require_POST
@transaction.atomic
def add_review(request, book_id):
    book = get_object_or_404(Book, pk=book_id)
    review = Review.objects.create(book=book, body=request.POST["body"])
    # Enqueue only after the surrounding transaction has committed.
    moderate_review.delay_on_commit(review.pk)
    notify_author.delay_on_commit(review.pk)
    return JsonResponse({"id": review.pk}, status=201)

Source: Tasks: database transactions and Performing actions after commit

Two consequences follow from how delay_on_commit() works. It does not return the task id, because nothing has been sent when the call returns; use plain delay() when you need the id and nothing depends on the commit. And if the transaction rolls back, the callback never runs, which is exactly what you want: no message for a row that does not exist. Under Django’s default autocommit (no surrounding atomic) the callback runs immediately. A task class that does not inherit DjangoTask (a custom base= that extends plain Task) lacks the method; inherit DjangoTask instead. With Celery older than 5.4, wrap the call yourself with transaction.on_commit(functools.partial(task.delay, pk)).

A test can assert both sides of the contract with captureOnCommitCallbacks():

Checking that nothing runs before the commit (catalog/tests.py)
from django.test import TestCase, override_settings

from .models import Book, Review


@override_settings(CELERY_TASK_ALWAYS_EAGER=True, CELERY_TASK_EAGER_PROPAGATES=True)
class ReviewViewTests(TestCase):
    def setUp(self):
        self.book = Book.objects.create(title="Dune")

    def test_view_enqueues_after_commit(self):
        with self.captureOnCommitCallbacks(execute=True) as callbacks:
            response = self.client.post(f"/books/{self.book.pk}/reviews/", {"body": "Great"})
        self.assertEqual(response.status_code, 201)
        self.assertEqual(len(callbacks), 2)
        self.assertIsNotNone(Review.objects.get().moderated_at)

    def test_nothing_is_enqueued_before_commit(self):
        with self.captureOnCommitCallbacks(execute=False):
            self.client.post(f"/books/{self.book.pk}/reviews/", {"body": "Great"})
        self.assertIsNone(Review.objects.get().moderated_at)

Source: TestCase.captureOnCommitCallbacks and Testing with Celery

Django’s TestCase wraps each test in a transaction that is rolled back, so on_commit callbacks never fire by themselves; captureOnCommitCallbacks(execute=True) runs them when the block exits.

Results: read them rarely, never inside a view

result = task.delay(…​) returns an AsyncResult; result.get(timeout=…) blocks until the worker is done and needs a result backend. Blocking a web worker on a Celery task throws away the point of using it, and by default Celery does not allow a task to wait synchronously for another task’s result. A page that needs the outcome should render now, store the task id, and let the browser poll a small endpoint or receive the result over a channel (Django Channels, described on its own page of this section, can push the completion to the client). Most tasks should write their outcome to the database themselves and set ignore_result=True.

Brokers and result backends

The broker transport is chosen by the scheme of CELERY_BROKER_URL. Celery’s documented comparison:

Broker Status Notes Monitoring (Flower, events)

RabbitMQ

Stable

The default transport, no extra package. Handles larger messages better than Redis; supports quorum queues (Celery 5.5 and later).

Yes

Redis

Stable

celery[redis]. Fast for many small messages, large messages can congest it. Has a visibility timeout (see below).

Yes

Amazon SQS

Stable

Fully managed and very scalable; lacks the remote-control commands.

No

Kafka, Google Pub/Sub, Zookeeper

Experimental

May work, but have no dedicated maintainers.

Kafka no, Pub/Sub yes

Source: Backends and brokers

Choose RabbitMQ when messages are large or delivery guarantees matter, and Redis when you already run it, messages are small and a lost message after a Redis failover is acceptable or covered by idempotent retries. Using Redis for the Django cache and as the Celery broker at once is common but couples two very different failure profiles; give them separate databases or instances. Redis as a Django cache is covered on Caching, and the broker side on RabbitMQ Queue Types, Consuming from RabbitMQ, RabbitMQ Reliability, Dead-Lettering and TTL, Delivery Guarantees and Kafka vs. RabbitMQ; for Redis as infrastructure see Redis persistence and Redis security. Celery does not make Kafka a good job queue; if you really want a log, use Kafka directly (Kafka consumers).

Choosing the transport by environment variable
export CELERY_BROKER_URL="redis://localhost:6379/0"          # local development
# production: a rediss:// or amqps:// URL injected by the platform or a secrets manager

Source: Using Redis and Using RabbitMQ

The Redis visibility timeout

With Redis, a message that is not acknowledged within the visibility timeout (one hour by default) is redelivered to another worker. That makes ETA, countdown and retry tasks whose delay exceeds the timeout run again, and again, in a loop. Either keep delays below the timeout, raise visibility_timeout in CELERY_BROKER_TRANSPORT_OPTIONS (and the matching result-backend option, which the Celery documentation asks you to set together), or schedule distant work from the database (django-celery-beat, or a model with a run_at column and a periodic task). RabbitMQ has no visibility timeout; its equivalent concerns are the consumer acknowledgement timeout and dead-lettering.

Raising the Redis visibility timeout (config/settings.py)
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 12 * 3600}  # seconds

Source: Redis: visibility timeout

Result backends

A result backend stores task state and return values. Redis is the usual fast choice (mind its memory limit and persistence); the rpc:// backend works with RabbitMQ; and for Django projects the django-celery-results package stores results through the ORM (CELERY_RESULT_BACKEND = "django-db") or the Django cache ("django-cache"). The ORM backend polls the database, which is expensive at volume; use it when results are few and you want to inspect them in the admin. In the scratch project django-celery-results 2.6.0 (2025-04-10; its classifiers stop at Django 5.2) installed and migrated on Django 6.1.1, and an embedded worker stored a successful task in TaskResult.

Storing results with the Django ORM (config/settings.py)
INSTALLED_APPS += ["django_celery_results"]   # note the underscores
CELERY_RESULT_BACKEND = "django-db"
Creating the result tables
python manage.py migrate django_celery_results

Source: django-celery-results

Celery used to support the Django ORM as a broker; that was removed in Celery 4.0 (2016). Do not use the database as a broker with Celery; the third-party django.tasks database backends exist for that role (see Celery or django.tasks?).

Retries

Networks fail, and many tasks call services that fail transiently. Celery retries in two ways. The explicit way, inside a bound task, calls self.retry(exc=exc, countdown=…), which raises Retry so the rest of the body does not run. The declarative way is the autoretry_for option, which wraps the body in the same try/except for the listed exception classes. Combined with retry_backoff it gives exponential delays with jitter, which is what you want for a service that is struggling:

Option Meaning (Celery defaults)

autoretry_for

Exception classes that trigger an automatic retry. Default: none.

dont_autoretry_for

Exceptions that match autoretry_for but must not be retried.

max_retries

Retries before giving up; None means forever. Default 3.

default_retry_delay

Seconds between retries when no backoff is set. Default 180 (three minutes).

retry_backoff

True doubles the delay (1 s, 2 s, 4 s …); a number is the first delay in seconds and the factor. Default off.

retry_backoff_max

Cap on the backoff delay in seconds. Default 600.

retry_jitter

Randomizes each delay between zero and the computed value so retries do not all fire together. Default on.

retry_kwargs

Extra arguments for the internal retry() call, for example {"max_retries": 5}.
A notification task with automatic retries (catalog/tasks.py)
@shared_task(
    bind=True,
    autoretry_for=(ConnectionError, TimeoutError),
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True,
    max_retries=5,
    acks_late=True,
    ignore_result=True,
)
def notify_author(self, review_id):
    review = Review.objects.select_related("book").get(pk=review_id)
    if review.notified_at is not None:  # already done: safe to run again
        return "skipped"
    send_mail(
        subject=f"New review for {review.book.title}",
        message=review.body,
        from_email=None,
        recipient_list=["author@example.com"],
    )
    # Mark after sending: a crash in between means a duplicate email (at-least-once).
    Review.objects.filter(pk=review_id, notified_at__isnull=True).update(
        notified_at=timezone.now()
    )
    return "sent"

Source: Automatic retry for known exceptions and Sending email

Retry only exceptions that are transient (ConnectionError, TimeoutError, an HTTP 503). Retrying Exception retries bugs, DoesNotExist after a rollback and validation errors, all of which will fail the same way. A test can simulate the flaky service by patching send_mail and checking the outcome after the retries:

Testing retries (catalog/tests.py)
from unittest import mock

from django.test import TestCase, override_settings

from .models import Book, Review
from .tasks import notify_author


@override_settings(CELERY_TASK_ALWAYS_EAGER=True, CELERY_TASK_EAGER_PROPAGATES=False)
class RetryTests(TestCase):
    def test_transient_errors_are_retried(self):
        review = Review.objects.create(book=Book.objects.create(title="Dune"), body="Great")
        attempts = []

        def flaky_send_mail(*args, **kwargs):
            attempts.append(1)
            if len(attempts) < 3:
                raise ConnectionError("SMTP server unreachable")
            return 1

        with mock.patch("catalog.tasks.send_mail", side_effect=flaky_send_mail):
            notify_author.delay(review.pk)
        self.assertEqual(len(attempts), 3)
        review.refresh_from_db()
        self.assertIsNotNone(review.notified_at)

Source: Testing with Celery

The scratch run passed (Ran 6 tests … OK). One observed detail is worth knowing: with CELERY_TASK_EAGER_PROPAGATES = True the eager-mode retry did not re-run the task; the Retry exception escaped from delay(). With propagation off, eager mode re-executes the retry immediately and ignores the delay, which is what the test above relies on. This is one of the ways eager mode differs from a real worker.

Two limits protect the worker pool from tasks that never finish: CELERY_TASK_TIME_LIMIT kills the task process, and CELERY_TASK_SOFT_TIME_LIMIT raises SoftTimeLimitExceeded inside the task so it can clean up. Set both on every project; a task that blocks indefinitely eventually stops a worker from doing anything else.

Idempotency and delivery guarantees

A queue between two processes cannot give exactly-once execution. Celery’s documented behaviour is:

  • By default the worker acknowledges the message before it runs the task. A task whose worker crashes mid-way is lost (at-most-once), but a task that has started is never started again.

  • With acks_late=True (per task) or CELERY_TASK_ACKS_LATE = True (global) the message is acknowledged after the task returns. If the worker dies mid-way, the message is redelivered (at-least-once) and the task runs again.

  • Even with late acknowledgement, a task whose pool process is killed by a signal or calls sys.exit() is acknowledged, to avoid loops caused by segmentation faults or the OOM killer. CELERY_TASK_REJECT_ON_WORKER_LOST = True opts in to redelivery in that case, with the loop risk that implies.

  • Brokers can also redeliver by themselves (the Redis visibility timeout above, a RabbitMQ consumer losing its connection).

So the rule is: enable acks_late only for tasks that are idempotent, and make every task you retry idempotent. The general theory, including effectively-once processing, is on Delivery Guarantees and Messaging Patterns in Practice; the Django-specific tools are:

  • Conditional update as a claim. Review.objects.filter(pk=pk, moderated_at__isnull=True).update(…) returns the number of rows changed. Only the first execution sees 1; a duplicate sees 0 and does nothing. This is moderate_review above, and it is race-free because the check and the write are one SQL statement.

  • State columns. notified_at, processed_at and status fields make the task a state machine whose steps can be re-entered.

  • Database constraints. A UniqueConstraint on (order_id, kind) turns a duplicate insert into an IntegrityError that the task catches and treats as success (see Constraints, Indexes and Composite Keys).

  • Idempotency keys for outside effects. Passing a key derived from the row (such as review-42-notify) to a payment or email provider that supports one lets the provider drop the duplicate.

  • Order the side effect and the marker deliberately. In notify_author the email is sent first and the marker written second, so a crash between them sends a second email (at-least-once). Marking first would lose the email instead (at-most-once). Pick the failure that costs less; for something that must happen exactly once, use a transactional outbox instead of a plain task.

A test proves the property by running the task twice:

Running a task twice (catalog/tests.py)
from django.core import mail
from django.test import TestCase, override_settings

from .models import Book, Review
from .tasks import moderate_review, notify_author


@override_settings(CELERY_TASK_ALWAYS_EAGER=True, CELERY_TASK_EAGER_PROPAGATES=True)
class IdempotencyTests(TestCase):
    def test_tasks_are_idempotent(self):
        review = Review.objects.create(book=Book.objects.create(title="Dune"), body="Great")
        self.assertTrue(moderate_review.delay(review.pk).get())
        self.assertFalse(moderate_review.delay(review.pk).get())
        notify_author.delay(review.pk)
        notify_author.delay(review.pk)
        self.assertEqual(len(mail.outbox), 1)

Source: Task.acks_late and Reject

Reasons a task fails that have nothing to do with the broker are covered in Common pitfalls: stale arguments, a missing row because the transaction had not committed, and non-serializable arguments.

Periodic tasks with celery beat

celery beat is a scheduler process: it reads a schedule and publishes a task message to the broker whenever one is due; the ordinary workers execute them. With the CELERY namespace the schedule is the Django setting CELERY_BEAT_SCHEDULE (the lower-case name is beat_schedule). The key task is the registered task name, schedule is a number of seconds, a timedelta or a crontab.

The beat schedule (config/settings.py)
from datetime import timedelta

from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    "remind-unmoderated-every-morning": {
        "task": "catalog.tasks.remind_unmoderated",
        "schedule": crontab(hour=7, minute=30),
    },
    "count-reviews-every-15-minutes": {
        "task": "catalog.tasks.count_reviews",
        "schedule": timedelta(minutes=15),
        "args": (1,),
        "options": {"expires": 600},
    },
}

Source: Periodic tasks

crontab() accepts minute, hour, day_of_week, day_of_month and month_of_year, each as a number, a list string ("0,30"), a range ("8-17") or a step ("*/15"), and Celery also has solar schedules (sunrise, sunset) for the rare case that needs them. A timedelta schedule counts from when beat starts, so the first run happens one interval after start, not at a round clock time. Give a periodic message an expires option so that a backlog does not make a worker run fifteen stale copies of a job after an outage.

Running beat and a worker
celery -A config beat -l INFO
celery -A config worker -l INFO

Time zones: periodic schedules use UTC by default, but a Django project’s TIME_ZONE is used when Celery is configured from Django settings, and a Celery-only zone can be set with CELERY_TIMEZONE. With USE_TZ = True (the default since Django 5.0) the schedule and your stored datetimes agree.

Beat’s rules, from the Celery documentation:

  • Run exactly one beat process for a given schedule. Two beat processes publish every job twice. Put beat in its own deployment or container with one replica; do not start it inside every worker.

  • celery worker -B embeds beat in a worker. It is convenient for development, but the documentation does not recommend it for production because it multiplies with every worker you start.

  • The default scheduler keeps its last-run times in a local file (celerybeat-schedule), so the directory must be writable; -s chooses another location. A container that loses the file on restart loses the last-run times too.

  • Like cron, jobs can overlap if one run outlasts its interval. Use a lock or an idempotent claim if that matters.

In the scratch project a real beat process was started against a schedule of one entry with a two-second interval; its log showed the expected output:

[… INFO/MainProcess] beat: Starting...
[… INFO/MainProcess] Scheduler: Sending due task demo (catalog.tasks.remind_unmoderated)
[… INFO/MainProcess] Scheduler: Sending due task demo (catalog.tasks.remind_unmoderated)

Source: Starting the scheduler

Database-backed schedules: django-celery-beat

The django-celery-beat extension stores the schedule in the database and adds admin pages to create, disable and edit periodic tasks at runtime, which suits schedules that operators change. You add django_celery_beat to INSTALLED_APPS, migrate, and start beat with its scheduler class.

Using the database scheduler (config/settings.py)
INSTALLED_APPS += ["django_celery_beat"]   # note the underscores
CELERY_BEAT_SCHEDULER = "django_celery_beat.schedulers:DatabaseScheduler"
python manage.py migrate django_celery_beat
celery -A config beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler

Source: Using custom scheduler classes

Compatibility needs care. django-celery-beat 2.9.0 (2026-02-28) declares Django<6.1,>=2.2 and classifies Django up to 6.0, so a normal install onto Django 6.1 makes the resolver downgrade Django to 6.0.x (that is what uv did in the scratch project) or report a conflict. Installed by force with --no-deps on Django 6.1.1, the app passed manage.py check and migrate, and beat logged DatabaseScheduler: Schedule changed. and created the three periodic-task rows (the two schedule entries plus the built-in celery.backend_cleanup). That is evidence that it works, not a support statement; check the project’s release notes before relying on it with 6.1. Changing time-zone settings does not reset a database schedule; the Celery documentation shows resetting last_run_at manually.

Monitoring with Flower

Flower is the web monitor and administration tool the Celery documentation recommends: it shows workers, task history and arguments, graphs and statistics, and lets you inspect workers, shut them down or restart them and change their pool size. It listens for Celery events, so workers must send them (start them with -E or set CELERY_WORKER_SEND_TASK_EVENTS = True), and the broker transport must implement events (RabbitMQ and Redis do; SQS does not).

Installing and running Flower
python -m pip install flower
celery -A config flower --port=5555

Source: Flower: getting started and Monitoring and management guide

Flower can run any command and see task arguments, so treat it as an administrative surface:

  • Do not expose it to the internet. Bind it to localhost or a private network (--address=127.0.0.1), and put a reverse proxy with TLS in front if it must be reachable.

  • Enable authentication: --basic-auth takes comma-separated user:secret pairs, and every option is also available as an environment variable with the FLOWER_ prefix (FLOWER_BASIC_AUTH, FLOWER_PORT, …), which is how to keep the credentials out of the command line; read the value from your secrets manager. OAuth providers (Google, GitHub, Okta, GitLab) can be configured instead, with --auth restricting which email addresses are allowed.

  • In Flower 2.2.0 the JSON API is disabled when authentication is off. The scratch run started Flower with no authentication: the home page and /healthcheck answered, while /api/workers returned 401 with the message that the FLOWER_UNAUTHENTICATED_API environment variable is required. Setting that variable enables the API and lets browsers on any origin read responses (CORS), so do not set it on a reachable instance.

  • --persistent=True --db=<file> keeps Flower’s state across restarts, max_tasks bounds its memory (default 100000) and purge_offline_workers removes dead workers from the list. Flower also exposes Prometheus metrics (see RabbitMQ Operations, Monitoring and Security for the broker side of monitoring).

Flower was run only against the in-memory broker in the scratch project, so the live worker and queue views were not exercised. For Django-side visibility, the django-celery-results admin pages (task id, status, result, traceback) are a simpler alternative with no extra service, at the cost of database writes per task.

Source: Flower: configuration and Flower: API reference

Testing Celery code

Celery’s own testing guide states the preferred approach: mock the task call in the code that enqueues, and unit-test the task body as a plain function, because a task is like a view that should only describe how to run the action as a task. That is why the tasks above are thin and why notify_author can be called directly with notify_author.run(pk) or notify_author.apply(args=(pk,)).

Eager mode (CELERY_TASK_ALWAYS_EAGER = True) runs delay() synchronously in the calling process. It is very handy for Django test cases and for the examples on this page, but the Celery documentation says explicitly that it is by definition not suitable for unit tests: you test an emulation of a worker, and it differs from a real one (no serialization round trip, no broker, no acknowledgement, different retry behaviour, and by default no result stored unless CELERY_TASK_STORE_EAGER_RESULT is on). Set CELERY_TASK_EAGER_PROPAGATES = True so exceptions surface in tests, except where you test retries (see above).

For a faithful check, Celery ships celery.contrib.testing.worker.start_worker, which runs a real worker in a thread; the pytest-celery plugin (Docker based) covers smoke tests against real brokers.

Running a real worker in a thread against the in-memory broker
import django

django.setup()  # after setting DJANGO_SETTINGS_MODULE

from celery.contrib.testing.worker import start_worker

from catalog.tasks import count_reviews
from config.celery import app

with start_worker(app, perform_ping_check=False, pool="solo"):
    result = count_reviews.delay(1)
    print(result.get(timeout=10), result.state)
1 SUCCESS

Source: Testing with Celery

The scratch run used the project’s SQLite file after migrate. A worker thread has its own database connection, so inside a TestCase (whose data lives in an uncommitted transaction) it may not see the test’s rows; use TransactionTestCase for such tests.

Running Celery in production

  • Processes. One celery worker deployment per queue family, one celery beat replica, optionally Flower. Run them under a supervisor (systemd, a container orchestrator); the Celery daemonization guide covers init systems, and Docker and Kubernetes deployments are on this site’s container pages. Windows is not supported by Celery (since 4.0), so use WSL or a container for development there.

  • Queues and routing. Send slow or sensitive tasks to their own queue so a flood of one kind cannot starve the others: CELERY_TASK_ROUTES = {"catalog.tasks.notify_author": {"queue": "emails"}} and start a worker with -Q emails (the scratch project confirmed the router sends notify_author to emails and everything else to the default celery queue).

  • Concurrency and prefetch. -c / CELERY_WORKER_CONCURRENCY sets the number of pool processes (default: the CPU count). The default worker_prefetch_multiplier is 4 messages per process; for long tasks set it to 1 so the first worker to start does not hoard the queue. --max-tasks-per-child recycles processes to contain leaks.

  • Database connections. Each pool process holds its own Django connections, and processes are forked after Django is set up. Django 5.1 added built-in connection pooling for PostgreSQL; Celery 5.6 closes Django’s pools before forking so that workers do not hit psycopg_pool.PoolTimeout (fixed in 5.6.0). Do not share a connection across processes. Size max_connections for workers x concurrency plus the web tier (see PostgreSQL and Database Backends).

  • Secrets. The broker URL, result backend URL and Flower credentials come from the environment or a secrets manager, never from the repository or the image. Use rediss:// / amqps:// with TLS when the broker is not on a private network.

  • Python version. Celery 5.6 lists CPython 3.9-3.13 as supported, while Django 6.1 needs 3.12 or newer. The examples ran on 3.13; test Python 3.14 before relying on it, because the Celery 5.6 release notes do not list it.

  • Graceful shutdown. A warm shutdown (SIGTERM) lets running tasks finish; worker_soft_shutdown_timeout adds a window before a cold shutdown, which matters when a Redis visibility timeout is long.

  • Settings drift. The worker imports the same Django settings as the web process; deploy them together so a renamed model field or task argument does not meet an old worker. Keep task signatures backwards compatible for one release.

Source: Workers guide, Routing tasks and Optimizing

Celery or django.tasks?

Django 6.0 added a Tasks framework in django.tasks (see Tasks Framework). It defines the contract (a @task decorator, enqueue(), TaskResult, the TASKS setting, on_commit guidance) but, in the words of the documentation, it does not provide a worker mechanism: execution is left to a backend and a process outside Django, and the two built-in backends (ImmediateBackend, the default, and DummyBackend) are for development and testing only. Production use therefore needs a third-party backend such as those of the django-tasks package (0.12.0, 2026-02-06, which also backports the API to Django 5.x). Celery is that kind of mature engine, with its own API.

Concern Celery django.tasks

Shipped with Django

No; a separate package and set of services.

Yes, since 6.0 (django-tasks backports it to older Django).

Worker and queue

Built in: worker, broker, prefork pool, many transports.

Not built in; the backend supplies them. Built-in backends do not run anything in the background.

Retries and backoff

autoretry_for, retry_backoff, max_retries.

Not part of the documented API; TaskContext.attempt reports the attempt number, the backend or your code decides.

Periodic tasks

celery beat, django-celery-beat.

Not part of the framework; use cron, a scheduler or the backend.

Routing and priorities

Queues, routes, exchanges, priorities (transport dependent).

queue_name and priority (-100 to 100) if the backend supports them; run_after for deferral.

Workflows

Canvas: chains, groups, chords.

None.

Result access

AsyncResult and result backends.

TaskResult and get_result(), if the backend supports it.

Arguments

JSON by default; configurable.

JSON round-trip enforced; arguments must keep their type.

Async API

Synchronous delay(); no native async API.

aenqueue() and aget_result().

Monitoring

Flower, events, Prometheus.

Whatever the backend offers.

Swappability

Tied to Celery’s API.

Task code stays the same when the backend changes.

Source: Django’s Tasks framework and Tasks reference

For the same job the two APIs look similar. The Django version, run in the scratch project on the default ImmediateBackend (which executes synchronously, so the result is available at once):

The same job with django.tasks (catalog/jobs.py)
from django.tasks import task


@task(queue_name="default", priority=0)
def count_reviews_dj(book_id):
    from .models import Review

    return Review.objects.filter(book_id=book_id).count()
Enqueueing it after commit
from functools import partial

from django.db import transaction

transaction.on_commit(partial(count_reviews_dj.enqueue, book.pk))
result = count_reviews_dj.enqueue(book.pk)
print(result.status, result.return_value)   # SUCCESSFUL 1 with ImmediateBackend

Source: Enqueueing Tasks and The task decorator

A practical way to choose:

  • Choose django.tasks for a new project whose background work is simple (send an email, resize an image), when you want the least infrastructure and a standard API, and when a backend that fits your queue exists. You can start with the ImmediateBackend in development and swap the backend later without touching task code.

  • Choose Celery when you need retries with backoff, scheduled jobs, routing to several queues, canvas workflows, rate limits, mature monitoring (Flower) and years of production experience, or when the project already runs it.

  • You can use both. They have different decorators and settings and do not interfere. A realistic migration is to leave existing Celery tasks alone, write new simple tasks with @task, and move over only when a django.tasks backend covers the features you rely on. Whichever you use, apply the same rules: enqueue after commit, pass identifiers, make tasks idempotent.

  • If you only need a clock, a management command called from cron or a platform scheduler is simpler than either (see Management Commands and Shell).

Both frameworks defer to the same message-queue fundamentals; for the models behind queues, topics and consumers see Messaging Models and Patterns.

Common pitfalls

  • The worker cannot find the row. The task ran before the commit. Use delay_on_commit() (Celery 5.4 and later) or transaction.on_commit().

  • The worker writes stale data. A model instance (or a snapshot of fields) was passed as an argument. Pass the primary key and re-fetch.

  • kombu.exceptions.EncodeError when calling delay(). The argument is not JSON-serializable (a model instance, a queryset, a set). Pass primary keys or plain values; do not switch to pickle.

  • A task runs twice. At-least-once delivery (late acknowledgement, a visibility timeout, a retry after a partial success). Make it idempotent.

  • A task never runs. No worker is listening on the queue the task was routed to, the task module is not imported by autodiscovery (check the [tasks] list on worker start), or the task name in CELERY_BEAT_SCHEDULE is wrong.

  • Everything looks fine in tests and breaks in production. Eager mode skipped serialization, the broker and acknowledgement.

  • Blocking result.get() in a view or inside another task. Store the id and poll, or write the outcome to the database.

  • Two beat processes. Every periodic job runs twice. Run one.

  • Lost or doubled work after a Redis restart. Redis is not durable by default; enable persistence and prefer RabbitMQ for work that must not be lost.

  • Settings typos. A misspelled CELERY_ key is ignored without a warning (the scratch project’s CELERY_TASK_ALWAYS_EAGR simply did nothing), so confirm the value Celery actually uses with app.conf.task_always_eager in python manage.py shell.

Where older material is outdated

Tutorials and books written for Celery 3 or 4 and Django 1.x-2.x teach patterns that have changed:

  • Separate libraries. django-celery was needed before Celery 3.1. Since 3.1 Django is supported out of the box; the djcelery package supports Celery 4.0 and below only.

  • from celery import task. The celery.task module was deprecated in 4.0 and removed in 5.0. Use from celery import shared_task.

  • Upper-case CELERY_ names without a namespace. Celery 4.0 introduced lower-case setting names. Django projects keep upper-case names by loading with namespace="CELERY"; celery upgrade settings --django converts an old settings file. Renamed examples: CELERY_ALWAYS_EAGER is now CELERY_TASK_ALWAYS_EAGER, and CELERY_RESULT_BACKEND stays while BROKER_URL became CELERY_BROKER_URL.

  • Options after the sub-command. Celery 5.0 rewrote the command line with Click: global options such as --app now come before the sub-command (celery -A config worker, not celery worker -A config).

  • delay() right after save(). Older Django code could not wait for the commit; transaction.on_commit arrived in Django 1.9 and delay_on_commit() in Celery 5.4.

  • Passing model instances and pickle. The pickle default ended with Celery 4.0; passing instances is a race condition whichever serializer is used.

  • The Django ORM as a broker was removed in Celery 4.0; the ORM stays usable as a result backend through django-celery-results.

  • The AMQP result backend was removed in Celery 5.0 (use rpc:// or another backend).

  • Python 2 and Python 3.7 or older. Celery 5.0 dropped Python 2.7 and 3.5, 5.4 dropped 3.7, 5.6 dropped 3.8.

  • Monitoring with Django-Admin tools. Flower obsoletes the old Django-Admin monitor, celerymon and the curses monitor in practice.

  • Classic mirrored RabbitMQ queues. Celery supports quorum queues since 5.5 and documents migrating from mirrored classic queues; with quorum queues autoscaling and prefetch reduction do not work, and ETA and countdown use native delayed delivery.

What changed recently

Version Change

Celery 5.0

Python 3 only; celery.task module and the AMQP backend removed; Click-based command line.

Celery 5.4

delay_on_commit() and the DjangoTask base class; zoneinfo replaces pytz.

Celery 5.5

Quorum queue support for RabbitMQ; Redis broker stability fixes (Kombu 5.5); Python 3.13 support; Django 2.2.28 is the minimum, --skip-checks flag added.

Celery 5.6 (5.6.3, 2026-03-26)

Python 3.9 to 3.13 supported, 3.8 dropped; Django connection pools are closed before forking; Redis backend and cold-shutdown improvements; a fix that prevents broker credentials leaking.

Django 5.1

Built-in PostgreSQL connection pooling (OPTIONS["pool"]) that Celery’s Django integration now cooperates with.

Django 6.0

The Tasks framework, django.tasks, with @task, enqueue(), TASKS and the Immediate and Dummy backends.

Django 6.1

@task accepts arbitrary keyword arguments for custom task classes; Celery 5.6.3 imported and ran on 6.1.1 in the scratch project although Celery’s pages still state support in terms of Django 2.2 or newer.

Flower 2.2.0

The API is off unless authentication is configured or FLOWER_UNAUTHENTICATED_API is set.

Source: What’s new in Celery 5.0, 5.4, 5.5, 5.6, 4.0, Django 6.1 release notes, 6.0 and 5.1

References