We had an outage on Friday 15th March caused by an innocent-looking change, where the “related name” of a Django model field was renamed.
We had wrongly assumed this didn’t require any schema changes to our Postgres 9.5 database, but it did: it dropped and recreated the foreign-key constraints on the table.
You can reproduce this behaviour by running sqlmigrate
on a migration created after renaming a
related_name
attribute:
$ ./manage.py sqlmigrate $APP_LABEL $MIGRATION_NUMBER
which yields something like:
BEGIN;
--
-- Alter field foo on bar
--
SET CONSTRAINTS "bar_foo_id_xxx_fk" IMMEDIATE;
ALTER TABLE "bar" DROP CONSTRAINT "bar_foo_id_xxx_fk";
ALTER TABLE "bar" ADD CONSTRAINT "bar_foo_id_xxx_fk"
FOREIGN KEY ("foo_id") REFERENCES "foo" ("id")
DEFERRABLE INITIALLY DEFERRED;
COMMIT;
Adding a foreign-key constraint requires an ACCESS EXCLUSIVE
lock on the table,
blocking SELECT
queries. For large tables, adding the constraint may take a
while, which can lead to major operational problems as queries queue up. This is
what happened to us.
FYI, this behaviour in Django is inadvertent: there is an open bug in Django 2.1.x and below: #25253,
There isn’t an good workaround for this problem. The best approach we know of is to
update an already-applied migration file to reflect the new related name. Since
such a migration is already applied, no SQL will be run against your schema when
this change deploys and Django won’t pick up any changes when makemigrations
is run.
During our next internal blitzday (where we work on our tooling, dependencies and general codebase health), we’ll see if we can submit a patch to Django to fix this, to avoid anyone else getting caught out like we did.
- Estimating cost per dbt model in Databricks
- Automating secrets management with 1Password Connect
- Understanding how mypy follows imports
- Optimizing AWS Stream Consumer Performance
- Sharing Power Updates using Amazon EventBridge Pipes
- Using formatters and linters to manage a large codebase
- Our pull request conventions
- Patterns of flakey Python tests
- Integrating Asana and GitHub
- Durable database transactions in Django
- Python interfaces a la Golang
- Our in-house coding conventions
- Recommended Django project structure
- Using a custom Sentry client
- Improving accessibility at Octopus Energy
- Django, ELB health checks and continuous delivery
- Organising styles for a React/Django hybrid
- Testing for missing migrations in Django
- Hello world, would you like to join us?