Branching by Feature

Introduction

Now that we’ve released Sage 300 Online as well as the on-premise Sage 300 ERP 2014, we need to change the way we develop features going forwards. We would like to develop features and frequently deploy these to Sage 300 Online so that customers can take advantage of these as soon as possible. Plus we would like to start including more features in our Product Updates.

This means that we have two separate products that would like to release features out of the same source code on their own timelines.

This article describes some of the issues with doing this and describes some aspects of the procedures we are following. In some cases there are additional benefits and in some cases there is additional work.

Operating SaaSy

Generally we are switching from releasing big bang product versions every year or so, to releasing updates on a much frequent basis. I’ve blogged about this before in these articles: SaaSifying Sage and SaaSifying Accpac. This article is really just covering one aspect of this process, namely around how we now use our source control system. This is one aspect of a continuous deployment pipeline. It also involves the DepOps team.

Doneness Criteria

A key ingredient into making this all work is having good “Doneness Criteria”, so that when agile stories are completed they are fully done, including all testing, automation, documentation, etc. So that when a “done” story is included in the release it really works. The closer you can come to this the better. If you aren’t very close then you are going to need a lot of integration/regression testing steps between these minor small feature releases to ensure the quality of the product. An important aspect of this is to have good automated tests to find integration problems and reduce manual regression testing.

Source Code Branching

Source code control systems have the concept of branching, where you can create a branch for a feature and then go through the agile process developing the feature and when you are all done then you merge the branch back into the product. This is a very simple branching strategy which works great if you are developing one feature at a time. However in reality there is a lot of work going on simultaneously including multiple features being developed at once, sustainment work (bug fixing) and infrastructure work (like the 64 Bit version).

Modern source control systems like Subversion and Git provide very powerful features to easily create branches and to incorporate changes from other branches and finally merge branches back together. We’ve used Subversion for some time now and have a lot of source code and history stored here. But for new projects we’ve been creating them in Git, partly due to the more powerful branching features.

To work effectively there are usually quite a few branches, but hopefully not as many as on the fractal tree below.

So what sort of branches do we need? Potentially we could create a lot of branches in the Source Code. The main root of all the branches is called the trunk and we always want the trunk to be in a releasable state. Perhaps we can have a development branch, and a regression branch. Then from the development branch we fork off the individual features as separate branches. Then when we are happy with them we merge them into the regression branch for more testing and when this is competed they are merged into the trunk.

Generally this works well for one product, say a web product where the DevOps team controls what gets merged from the regression branch into the trunk. However when you have multiple products derived from the same source tree this can be quite complicated.

Another danger of too much branching is that the longer a branch lives, the further it can get from the trunk due to work done on other branches and then merging back into a main branch and the trunk gets more difficult. This can be alleviated by merging changes done elsewhere into your branch, so you have smaller merges along the way rather than a bit difficult merge at the end. Another advantage we have in Sage 300, is that it’s a very large product consisting of hundreds of DLLs, OCXs and EXEs. As a result different teams are often operating in quite different areas of the source code. However there will still be points of contention, one that is happening right now is multiple features being added to the reporting engine that is creating contention on that source code module.

So for a branching strategy we need to strike a compromise between keeping everything perfectly isolated and having a lot of gates for features to pass through versus keeping the management of the system down and reducing the difficulty in merging branches.

So our approach is to limit the number of branches. Trunk is the main branch that is released to production (i.e. customers). Nothing is developed directly on the trunk, everything needs to be developed on a feature branch. Before merging back into trunk, a feature must merge the trunk back into the feature first and the team developing the feature must do some testing (including running our full automated testing suite). Then the feature can be merged back into the trunk and the feature is available for either the Online or on-premise products to consume.

Consequences

Of course this has consequences on other components in the build pipeline. Now rather than just build the trunk with our set of build servers, we have to build all these branches. As part of the build we run a number of unit tests, but we also have a large set of automated tests that run on a separate set of servers, and now we want to run these automated tests on the branches.

Generally this increases the logistics of maintaining all these sets of servers. The build servers need to be configured on what branch to run and then the output of that build needs to be fed into the automated test servers to run all the tests.

Generally this all improves the quality and keeps the trunk in a releasable state since a lot of testing has been done before the feature is merged into the trunk.

For newer components like the Sage 300 Online home page and the Sage 300 ERP provisioning engine, these are built entirely using ASP.Net and then built and deployed using TeamCity. This simplifies the whole process and we can use a more sophisticated branching strategy in conjunction with DevOps where they have a release branch which only they control where they can have complete control on what they merge, build and deploy.

Summary

We’ve been putting this infrastructure in place for some time now and have it operating fairly smoothly. Now we will really start putting it into practice by releasing features frequently on two product lines. The main test that we’ve done it right is that it’s seamless to customers because we’ve been able to maintain high quality with all these processes and technologies in place.

Update 2014/10/04: This scheme has given us some problems. For an update on what we are doing as a result check out this article on the problems with branching by feature.