At Kraken Tech we have a large global development team with over 500 developers, the majority of which work on the same monolithic codebase comprising more than 4 million lines of Python code. We release new code over 100 times a day, running hundreds of thousands of tests in the process.
So how can we ensure high coding quality in a distributed team, with such frequent changes? And how can we make it easier for new joiners to slot right in and contribute?
Formatting
Not long after I joined the company, my first professional job as a programmer, we introduced Black. There was one giant refactor pull request to format the codebase with Black, admittedly something that would be much harder to do now that the codebase has grown to many times its original size. At the same time, we also introduced isort (which has now been superseded by Ruff in our setup). For me as a brand new dev, this was much needed and very helpful.
Before this change, I never really knew what the preferred structure was. In code reviews, you might get competing advice regarding “best practice” and general feedback on style. Having consistent formatting greatly reduces the mental load and decision-making needed when writing new code. Instead, you can just concentrate on the task at hand.
Not only that, it’s also much easier to find what you’re looking for. Ordered imports mean that you can immediately identify module imports and hence dependencies. Having consistent formatting also means it’s much easier to spot function arguments, doc strings, and the like.
We run Black and Ruff as part of our CI checks. This means that new code is not merged unless it conforms with our formatting standards. As you can configure your editor of choice to run formatting against changed files on save, this is not an additional hassle for developers. Everyone wins.
Linting
Another great tool at our disposal is linting. We use a variety of linters to reduce bugs and ensure developers conform to our conventions; we have many of them and we can hardly expect everyone to know them off by heart!
As the codebase is so large, we run some of our linters on changed files only. This allows us to introduce new rules and achieve gradual improvement, rather than putting the burden of fixing existing code on a single developer and causing a mass of merge conflicts. An alternative is to add ignores to the code base in one go and then rely on developers to fix things as they see it (silence-lint-error is a great package for that). Which approach is best depends on the situation.
Type checking
For type checking we use mypy. Type checking helps in finding bugs before they happen in production and hence prevent annoying errors at best, and costly outages at worst. We’ve had mypy enabled for many years. However, since Python isn’t traditionally a typed language, we had very few type hints in place in our early years and hence didn’t gain much value from running mypy.
This only changed when we introduced a custom linter, which forced all developers to add at the very least a return type to any function they touched. This is as mypy doesn’t actually type check a function unless there is at least one type annotation in the function signature; using the return type seemed like a good choice as a start. This has now been extended to enforce typing on all function arguments for changed functions.
Since that change, we’re actually seeing the benefits. It’s not only helpful in preventing bugs but also in simply understanding what the function does. Type hints are part of the documentation. We’ve also seen particular value when transitioning between legacy and new systems, as proper typing can make it clear which system is supported.
Of course, sometimes wrangling mypy can be a bit of a challenge, and there are particularly curious issues due to bugs in django-stubs, but the benefits by far outweigh the cons.
We track our missing type annotations and #type: ignores with a dashboard to
hold ourselves accountable; only that way do we know we’re making progress and
that our approach is working.
Making things better, one commit at a time
We also use custom linters within Fixit to enforce some of our other conventions. This could be around readability of code, documentation, or good practice around security. Examples are
- ensuring our test module paths mirror that of the module they are testing
- ensuring we’re correctly asserting mocks in tests
- prohibiting the use of deprecated functions
- enforcing naming conventions for certain Django field types
and many more. Whenever we spot a potential issue that can be prevented with a simple linter, we just add one.
As with mypy, we typically only check changed files. This means that each developer contributes a little bit to reducing technical debt with every pull request that touches non-conforming code. With our pull request conventions, this usually just means a small clean up commit on a module before introducing a functional change.
We have found this approach really works for us. We don’t overload our devs with what some may call boring clean up work. We also have a nice automatic way of ensuring the conventions we really care about are adhered to. As the linters live in the code, you can always go back to the original pull request introducing them to find a helpful discussion on why we introduced it or perhaps a link to a ticket or slack post.
Application layer linting
One of our developers, David Seddon, developed Import Linter which we use for ensuring our application conforms to prescribed layering, i.e. module imports are only allowed in a certain direction. This is an incredibly helpful tool for a large codebase like ours, as it prevents circular imports creeping in and ensures a clear separation of responsibilities. For more on this, you can read his blog post.
Conclusion
I hope you enjoyed this brief overview of some of the tools we use to manage to keep our code quality high. These tools are essential for a large team working on a single code base as we do at Kraken. We’re constantly evolving our tools to improve code quality as we grow. If you’re interested in helping us build better systems, check out our careers page.
- Estimating cost per dbt model in Databricks
- Automating secrets management with 1Password Connect
- Understanding how mypy follows imports
- Optimizing AWS Stream Consumer Performance
- Sharing Power Updates using Amazon EventBridge Pipes
- Our pull request conventions
- Patterns of flakey Python tests
- Integrating Asana and GitHub
- Durable database transactions in Django
- Python interfaces a la Golang
- Beware changing the "related name" of a Django model field
- Our in-house coding conventions
- Recommended Django project structure
- Using a custom Sentry client
- Improving accessibility at Octopus Energy
- Django, ELB health checks and continuous delivery
- Organising styles for a React/Django hybrid
- Testing for missing migrations in Django
- Hello world, would you like to join us?