A tale of working from trunk

Let me share with you a story about how we went from long lived feature/release branches to trunk based development, why it was really hard and whether this is something I would recommend you try.

Background

I’m familiar with three main approaches to code branching for a shared code-base:

  1. Long lived feature/release branches
  2. Short lived feature branches
  3. Trunk based development

Long lived feature/release branches

Most teams will start out using long lived feature/release branches. This is where each new project or feature branches from trunk and at a point where the branch is feature ready/stable then these changes are merged into trunk and released. The benefits of this approach is that changes are contained within a branch, so there’s little risk of non-finished changes inadvertently leaking into the main trunk, which is what is used for releases to production. The biggest downside to this approach, and why many teams move away from it, is the merging that has to happen, as each long lived feature branch needs to ultimately combine its changes with every other long lived feature branch and into trunk, and the longer the branch exists, the more it can diverge and the harder this becomes. Some people call this ‘merge hell’.

Short lived feature/release branches

Another version of feature branching is to have short lived feature branches which exist to introduce a change or feature and are merged (often automatically) into the trunk as soon as the change is reviewed and tested. This is typically done using a distributed version control system (such as GIT) and by using a pull request system. Since branches are ad-hoc/short-lived, you need a continuous integration system that supports running against all branches for this approach to work (ours doesn’t), otherwise it doesn’t work as you’d need to create a new build config every time you created a short lived feature branch.

Trunk Based Development

This is probably the simplest (and most risky) approach in that everyone works from and commits directly to trunk. This avoids the needs for merges but also means that trunk should be production ready at any point in time.

A story of moving from long lived feature/release branches to trunk based development

We have anywhere from 2-5 concurrent projects (each with a team of 8 or so developers) working off the same code base that is released to production anywhere from once to half a dozen times per week.

These project teams started out using long-lived feature/release branches specific to projects, but the teams increasingly found merging/divergence difficult – and issues would arise where a merge wasn’t done correctly, so a regression would be inadvertently released. The teams also found there would be manual effort involved in setting up our CI server to run against a new feature/release branch when it was created, and removing it when the feature/release branch was finished.

Since we don’t use a workflow based/distributed version control system, and our CI tools don’t support running against every branch, we couldn’t move to using short lived feature branches, so we decided to move to trunk-based development.

Stage One – Trunk Based Development without a Release Branch

Initially we had pure trunk based development. Everyone committed to trunk. Our CI build ran against trunk, and each build from trunk could be promoted right through to production.

Trunk Based Development without a Release Branch(1)

Almost immediately two main problems arose with our approach:

  1. Feature leakage: people would commit code that wasn’t behind a feature toggle which was inadvertently released to production. This happened a number of times no matter how many times I would tell people ‘use toggles!’.
  2. Hotfix changes using trunk: since we could only deploy from trunk, each hotfix would have to be done via trunk, and this meant the hotfix would include every change made between it and the last release (so, in the above diagram if we wanted to hotfix revision T4 and there were another three revisions, we would have to release T7 and everything else it contained). Trying to get a suitable build would often be a case of one step forward/two steps back with other unintended changes in the mix. This was very stressful for the team and often led to temporarily ‘code freezes’ whilst someone committed a hotfix into trunk and got it ready.

Stage Two – Trunk Based Development with a Release Branch

Pure trunk based development wasn’t working, so we needed some strategies to address our two biggest problems.

  1. Feature leakage: whilst this was more of a cultural/mindset change for the team learning and knowing that every commit would have to be production suitable, one great idea we did implement was TDC: test driven configuration. Since tests act as a safety net against unintended code changes (similar to double entry book-keeping), why not apply the same thinking to config? Basically we wrote unit tests against configuration settings so if a toggle was turned on without having a test that expected it to be on, it would fail the build and couldn’t be promoted to production.
  2. Hotfixing changes from trunk: whilst we wanted to develop and deploy from a constantly verified trunk, we needed a way to quickly provide a hotfix without including every other change in trunk. We decided to create a release branch, but not to release a new feature per say, but purely for production releases. A release would therefore involve deleting and recreating a release branch from trunk to avoid having any divergence. If an hotfix was needed, this could be applied directly to the release branch and the change would be merged into trunk (or the other way around), knowing that the next release would delete the release branch and start again from trunk. This alone has made the entire release process much less stressful as if a last minute change is needed for a release, or a hotfix is required, it’s now much quicker and simpler than releasing a whole new version from trunk, although that is still an option. I would say that nine out of ten of our releases are done by taking a whole new cut, whereas one or so out of ten is done via a change to the release branch.

Trunk Based Development with Release Branch(2)

Lessons Learned

It’s certainly been a ride, but I definitely feel more comfortable with our approach now we’ve ironed out a lot of the kinks.

So, the big question is whether I would recommend team/s to do trunk based development? Well, it depends.

I believe you should only consider working from trunk if:

  • you have very disciplined teams who see every-single-commit as production ready code that could be in production in a hour;
  • you have a release branch that you recreate for each release and can uses for hotfixes;
  • your teams constantly check the build monitors and don’t commit on a red build –  broken commits pile upon broken commits;
  • your teams put every new/non-complete feature/change behind a feature toggle that is toggled off by default, and tested that it is so; and
  • you have comprehensive regression test suite that can tell you immediately if any regressions have been introduced into every build.

Then, and only then, should you work all off trunk.

What have your experiences been with branching?