Fixtures, Serialization & Initial Data

Django looks for each fixture in three places: the fixtures directory of every installed app, every directory listed in the FIXTURE_DIRS setting, and the literal path named by the label. It loads every match with the requested name from all those places, in INSTALLED_APPS order. If the label has an extension (bookshelf.json) only that type is loaded; without one, every registered type is searched. Directory components in the label extend the search path, so loaddata shared/extra_tags looks for shared/extra_tags.* below each location. An absolute path switches the default search off. Use -v 2 to watch the search.

Source: How fixtures are discovered and FIXTURE_DIRS.

Because Django loads all fixtures of the same name, name your app fixtures distinctively (catalog_bookshelf.json, not data.json), otherwise a fixture from another app may be loaded together with yours.

Source: Compressed fixtures and Database-specific fixtures.

Three behaviours cause most surprises.

The second behaviour is what makes a natural-key fixture idempotent. Loading the same fixture twice into the example project reports Installed 6 object(s) both times and leaves two authors and two books in place.

Because signals receive raw=True, a receiver that touches related objects must skip raw saves; the related rows may not exist yet.

The receiver is connected in CatalogConfig.ready() (from . import signals). Loading the bookshelf fixture and then creating a book in the shell gave [('raw', '9780061054884'), ('raw', '9780807083697')] followed by ('normal', '1').

Source: How fixtures are saved to the database. See Signals for receivers in general.

When --natural-foreign is used, models that define natural_key() are serialized before models that do not. If a natural key itself depends on another object (a book identified by title and author, for instance), declare the dependency on the method so the other model is dumped first:

The dependencies attribute lists app-label.model labels (lowercase model name). Without it a fixture may contain a book before its author.

A fixture can still reference an object that appears later in the same file. Whether that loads depends on the foreign key. With the non-nullable Book.author it fails. With the nullable Author.mentor it works, because Django can save the row with NULL and fill the reference in once the target exists. This fixture lists Octavia Butler before the mentor she refers to:

loaddata handles this for you. When you call the deserializer yourself you must opt in with handle_forward_references=True and finish the job with save_deferred_fields():

Run against the fixture above, this restored Octavia Butler’s mentor. Without the flag the same data raised DeserializationError: Author matching query does not exist.. The same load against a fixture whose book comes before its (non-nullable) author fails at the end of the load with IntegrityError: …​ NOT NULL constraint failed: catalog_book.author_id.

Source: Natural keys and forward references.

Since Django 6.1 a subclass of a model that defines natural_key() can return an empty tuple () from it. The serializer then falls back to the primary key for that subclass, so dumpdata --natural-primary still writes the pk. The opt-out is new in 6.1; on 5.2 LTS a subclass inherits the parent’s natural_key().

Source: Django 6.1 release notes: Serialization.

serialize(format, queryset, **options) accepts any iterable of model instances, so a single object is passed as [obj]. Options: fields (a subset; note that the result cannot be loaded back for a model whose required fields are missing), use_natural_foreign_keys, use_natural_primary_keys, stream (write to a file-like object instead of returning a string), indent, and format-specific ones such as cls= for a custom JSON encoder. serializers.get_serializer("json") returns the class when you want to keep the serializer object and call getvalue() yourself. Serializing a model that inherits from a concrete parent writes only the child’s own fields, so serialize the parent rows as well when you want them in the output.

Source: Serializing Django objects and Subset of fields.

The same data in YAML (dumpdata catalog --natural-foreign --natural-primary --format yaml) and XML (dumpdata catalog.author --format xml --indent 2):

(The YAML output is trimmed to two objects.) Source: Serialization formats.

serializers.deserialize(format, data_or_stream, **options) returns an iterator of DeserializedObject instances. Each wraps a created but unsaved model instance in .object and its many-to-many data in .m2m_data; call .save() to write it. If the primary key is None, the manager’s get_by_natural_key() is used to look for an existing row, and a new row is inserted otherwise. The point of the iterator is that you can inspect each object and decide whether to save it.

Two options matter in practice:

Deserialization is not validation. DeserializedObject.save() writes the instance with raw=True semantics: full_clean() is not called, validators do not run, and save() of the model is bypassed. Run against [{"model": "catalog.tag", "pk": 99, "fields": {"name": ""}}], the loop saved a tag with an empty name even though the field is not blank=True. Never deserialize data from an untrusted source straight into the database; use a form, a DRF serializer or a Ninja schema (see Django REST Framework and Django Ninja) for input that crosses a trust boundary, and keep Django’s serializers for backups, fixtures and data you produced yourself. The Django serializers are not an API layer: they have no per-field output control beyond fields, no input validation and no nested representation.

Source: Deserializing data.

A serializer is a module that exposes Serializer and Deserializer classes, registered under a format name with the SERIALIZATION_MODULES setting. Both classes usually subclass an existing format. Since Django 5.2 every built-in format defines a Deserializer class rather than a function, which makes extending one easier. This example registers a compact format that flattens each object to {"id": 1, "name": …​}, which is handy for feeding a small front-end widget.

Because Deserializer is the stock JSON one, this format is write-only: it cannot be fed back to loaddata, since the output no longer has the model and fields keys. A format that must round-trip needs a matching Deserializer, as in the CSV example of the reference. To register a format at runtime rather than in settings, use serializers.register_serializer(format, serializer_module).

Source: Custom serialization formats.

Migrations run when the test database is created, so rows created by a data migration are available to tests (with the limitation described under fixtures in tests). Write it as a RunPython operation with a reverse function, and always obtain the model through apps.get_model(): the real catalog.models.Tag may have changed since the migration was written, while apps.get_model() gives the model as it was at that point in the history.

After the first command Tag.objects held ['fiction', 'non-fiction', 'reference']; the second removed them again. The historical model has no custom methods and no custom managers (unless use_in_migrations = True), which is why the example uses get_or_create on the plain manager. using(db_alias) keeps the migration correct in a multiple-database setup, and get_or_create makes it safe to rerun. See Migrations for RunPython, ordering and reversibility in depth.

A migration can also call call_command("loaddata", …​), but the fixture is loaded with your current models rather than the historical ones, so the migration breaks as soon as a later migration changes a model the fixture touches. Prefer RunPython for data that ships with the schema.

Source: RunPython and Data migrations.

post_migrate is sent after migrate finishes for each app, even when there was nothing to apply. Connect the handler in AppConfig.ready() with sender=self so it runs once per migrate, not once per app, and make it idempotent. This is the same mechanism Django uses to create permissions and content types. Handlers use the current models, so they suit rows that cannot be created inside a migration, such as groups attached to other apps' permissions.

After deleting all groups, python manage.py migrate recreated ['Editors'].

Source: post_migrate and AppConfig.ready().

For demo data, or data that differs per environment, write a command that creates or updates rows idempotently. It can be run from a deploy script, a Docker entrypoint or by hand, which makes it the most flexible option.

Unlike a fixture, this command goes through the model’s save() and signals, so it also runs your business rules. See Management Commands & Shell for the command machinery and call_command.

Source: Writing custom django-admin commands and update_or_create().

An initial superuser belongs in createsuperuser --noinput with DJANGO_SUPERUSER_USERNAME, DJANGO_SUPERUSER_EMAIL and DJANGO_SUPERUSER_PASSWORD taken from the environment or a secret store, not in a fixture. A fixture with a password hash is a credential in your repository. See Settings & Configuration for how to keep secrets out of source control.

Source: createsuperuser.

A TransactionTestCase does not wrap tests in a transaction; it flushes the tables after each test instead. That also removes the rows your data migrations created. In the example project a TransactionTestCase that ran after another one found 0 rows of the migration-seeded fiction tag. Setting serialized_rollback = True asks Django to restore the database from a snapshot taken when the test database was created; it brought the row back when that class ran first, but in the example project it raised an IntegrityError on django_content_type when it ran after another flushing class, and with the post_migrate group handler above it raised one on auth_group.name, because the flush re-sends post_migrate. So with TransactionTestCase, create the rows each test needs (with a factory, or a fixture through the fixtures attribute) rather than relying on migration data. Since Django 5.2, data from fixtures and from serialized_rollback is already available in TransactionTestCase.setUpClass().

Source: Rollback emulation and Django 5.2 release notes: Tests.