Weeknotes 2022/17 - Software Architecture: Splitting Up Large Projects

Most non-trivial real life production apps usually require an order of magnitude more planning and configuring than the simple templates from where most projects are started from. I spent the last week testing and planning out the final configuration which would be best suited for an Django app with Vue frontend. Here are some notes about the project and my process.

The Starting Point

I’m currently working on an app for taking and handling DJ Song Requests for Slipmat.io. It’s a fresh rewrite of an older version, with added features and improvements from the learnings of past ~6 years.

The old app created a simple song collection from an uploaded DJ playlist file, and then offered a frontend for listeners to browse that collection to make song requests and to vote for them. Artists could then browse the requests and mark them as played. The new app will have much more functionality, including full support for collaborative events and crowdsourced metadata editing to help enrich and improve the data.

Slipmat Requests v3

The previous versions of the app grew in a very typical organic way; “we should add feature X because user Y needs it”. There was never a bigger holistic plan, and the related technologies have evolved a lot during past few years as well. One of the biggest issues with the old app was that everything was bundled in a one giant lump. One Django app with all the backend models and a messy API, and one Vue app for everything in the frontend.

After the initial planning and rough architecture ideas I started by implementing the most basic MVP version of just the rough wireframe that I could imagine, and then iterated the final architecture from there.

The new app has three major frontend parts; the user app, the artist app, and the admin app. All of these apps talk to one common backend which consists of several smaller Django apps and two APIs; one for the site admins and another one for everything else. These apps will be running on three different domains.

Backend

All my Python and Django projects start nowadays from a template that has a Dockerfile and/or Docker compose file for running production. Simple projects only need one instance, but things like Celery (task queue) immediately bump the needed instances to two or more.

Django has always had the sites framework, and therefore also a pretty good support for splitting a bigger app up in smaller parts. One basic thing that most Django apps should do is to split all admin functionality (including URLs) into a different domain. That is one extra layer of security and it eliminates dumb accidents.

I want to keep the code as small as possible and the configuration easy so I decided to use just one settings file for all three sites, run the same instance for all domains, and use a small middleware to figure out which domain is currently active. I may change this in production later but for now it simplifies the development workflow quite a bit.

Frontend

I juggled the frontend app configuration a handful of times before settling down to the final version. The possibilities were basically either a multi-page Vite app, three Vite apps, or some kind of a monorepo as a combination of previous two.

I eventually went with the monorepo route, splitting the apps as three totally independent Vite projects, but also including one common folder of components and common scripts and used TypeScript and Vite configuration to allow easy importing and common configuration for all three projects.

I’m using pnpm workspace to manage the dependencies which allows me to easily keep everything in sync but also develop and deploy each of the apps individually if needed.

I’m still not 100% certain on how much code I should share for example between the user and artist app initialization and collection browsing. Obviously much of the functionality will be almost identical but there’s a tradeoff in writing super flexible components versus just copying simple components between the apps. This is one common use case where headless components are super useful and I’m going to be using that ideology a lot.

Development

Splitting the frontend apps exploded the complexity of development in many ways. If you need to run the apps concurrently, you need to somehow distinguish between which app is which, either by running them on fixed ports (for example port 3000 is always the main app, port 3001 is the artist app, and port 3002 is the admin app) or by setting up local domains with a hosts file and then setting up local SSL certificates as all of the apps use things like websockets which only work non-secured on localhost domain.

I started with the latter approach and used mkcert to create the needed certificates but I’m not yet sure if this is the way to go because Django runserver doesn’t have SSL support built in so the Django Channels runserver doesn’t support SSL. I worked around this by running a separate uvicorn process for the websockets but that isn’t very convenient.

In addition to the frontend and socket serving complexity, there’s currently no tooling for hosting several Vite apps with Django. I forked my own version of django-vite to handle this but it’s not likely to be something I’m keen to be maintaining for a long time. Again, need to test this thing out a bit more to get a better understanding of what would work best.

Production And CI/CD Pipeline

Production configuration was the simplest thing here. I’m using Docker and Docker compose for the configuration and Traefik as a proxy for the sites so adding more domains meant just adding a few more lines to the compose file.

During the first steps of development I use simple Fabric functions to deploy the site manually by typing fab deploy from the command line, and when the project matures I set this up with Github Actions for the deployments to happen automatically when merging to master or when merging master to stable, whatever works best for the project.

In any case this week was a good example of the complexity one doesn’t often think about when first planning things, even when the requirements are well known in advance. I probably suffer in my own projects from needing to do this all by myself, as within a team there’s at least a chance someone will catch things like this earlier in the process.