Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘ivy

Continuous Deployment

with 4 comments


I’ve talked a bit about SaaSifying Accpac and SaaSifying Sage. Today I’m going to talk a bit more about continuous deployment which is one of the end goals of these themes. This is the notion that there is an automated process that will take code from the programmer, build a product, test it and then deploy it so that customers are running it immediately. If you’ve worked in software development, this can be a scary notion, since you’re lucky if programmers check in something that compiles, let alone something you want customers running without a lot of checks and balances. The key to continuous deployment is automating many manual processes that used to be very repetitive,  time consuming and error prone.

A key to successful continuous deployment is to always be challenging the status-quo. Don’t accept that something absolutely requires manual intervention. Work hard to eliminate all gate-keepers. In a way continuous deployment is a mindset and process where you are always looking to improve your automated processes, eliminate manual processes and get features to customers as soon as the programmer commits their changes. Part and parcel with this is breaking major functionality down into many small features that are all deployed as they are done. Breaking things down into very small pieces is the key to success and the key to speeding up development. Implementing things as one big piece tends to be hard, complex and leads to customer problems.

Building the Product

Generally with modern source control systems, programmers should only check things into the main trunk once they have completed something including unit testing, automated tests and manual testing. If they need intermediate check-ins then they should use a code branch and merge that in when they are finished. However with large development projects, you can still get conflicts and unexpected results. You want to find these as quickly as possible and fix them right away while things are fresh in your mind.

The programmer checking in code is the beginning of the continuous deployment process. If they have been successful in coding and testing a useful feature for your customers, then you would like this to be available to your customers in a very short order. I blogged on the desirability of doing this here.

Now you have a build server running somewhere, perhaps being controlled by a continuous integration system such as Jenkins. The system on this server will be scanning source control for any committed changes, as soon as it sees one it will get it and build it and run all unit tests. Further it has a tree of all the dependencies in the system (usually using Ivy) and will build all dependent modules and run their unit tests. If anything fails then a number of designated people will receive e-mails that the build is now failing. The person that checked in the changes will also be notified. It is now up to them to quickly fix things (the system can be configured to automatically roll back whatever they checked in, but usually the build system just keeps using the last good build).

The key point here is that you find out right away that you broke the build. This is in contrast to say doing nightly builds, where you don’t know until the next day, and there may have been many check-ins so it can be time consuming finding the real culprit. Plus it now interrupts whatever you were meant to do the next day.

Continuous Test Server

Usually beyond unit tests, you have a continuous test server that may be using Fitnesse to run Selenium UI based tests. These tests are too long to be unit tests and would slow the build server down. However you want builds to be regularly deployed to this server and the full set of UI based automated tests to be run. Perhaps after it runs the full suite it would re-deploy the latest build (if newer) and run all the tests again and just keep continuously doing this. If a test fails it would immediately notify the appropriate people and demand prompt action to resolve the problem either by fixing the problem, fixing the test or rolling something back. Again the goal here is to detect problems as quickly as possible and to force them to be resolved quickly, not allowing problems to pile up. Generally you want to keep your product in a releasable state as I blogged about here.

Automated Processes to Deploy

Next we want to start deploying for various users to start playing with the system. So we have a build that has passed the unit tests and the automated longer running tests, now we start deploying for review by various departments. Usually this will mean automatic deployment to a server that various interested internal parties know how to access, perhaps with a common URL. Perhaps the build is deployed to various manual testing computers. Anyway at this point the system can be played with by anyone that needs to review things like User Centered Design Analysts, Business Analysts, Product Managers, other interested developers, executives, etc.

This step is actually optional. If your unit tests and other automated tests are good enough and if your development organization is confident enough and has a good track record then this step can be skipped and you can deploy directly to customers. Eliminating this step is a primary goal of many continuous development teams. Sometimes it’s just a matter of keeping one last human checkpoint in place, but there are many organizations that deploy to the production servers several times a day without needing this step. This is the fastest way to immediate customer feedback on what you are doing.

Deploy to Customers

Now the fun part, deploying to the production servers for customers. You want this to be the same process as described above so that you know it will work properly. Sometimes you might deploy to one server and have it start serving a set up customers to ensure all is well before allowing the build to go to all servers. Key to this is the ability to seamlessly update the database and have the ability to roll back the changes including the database changes if things go wrong.

Database Maintenance

Usually when you deploy new features, there have to be database schema changes to support these. Depending on the type of database system you are using whether SQL or NoSQL the types of changes could be quite different. In the SQL world you might add some fields to some tables or add some new tables. Generally when you design your schema changes you need to be cognizant of the demands of continuous deployment. For a web site that needs to be up 24×7, you can’t deploy something that requires hours of data conversion. Generally you want a very simple database design to allow changes without a great deal of fuss. If you end up really needing a major conversion, then this will have to be planned and you need to measure the ROI of such a feature given the disruption to customers of even small outages. Another approach is to make the application smarter so perhaps it can handle old data and new data so that the system can still run as the conversion is in progress. This of course takes more work that simply shutting things down, but becomes the cost of doing business in the 24×7 SaaS world.

Besides providing scripts and/or programs to convert the database, you have to provide scripts and/or programs to un-convert the database. By the same token if something goes wrong, say users start having problems, then you need to roll back the database changes (and the newly deployed build), again without causing a huge service blackout. This side of things can be very challenging. These processes need to be a heavy focus for unit and other automated testing.


Not all changes that pass all the testing and have been approved by all departments end up delighting customers. Often some unforeseen usage causes the new features to greatly irritate your customers, who then tell you and the world all about their annoyance on Twitter, LinkedIn, Facebook, etc. At this point you may need to make the hard decision to roll-back the newly deployed build along with any artifacts like database changes.

Even though we hate doing this, we need to remember that customer satisfaction is job one and that if we keep annoying them, they will go to another service that won’t annoy them so much. All this is a part of being Agile, customer focused and applying validated learning. Basically learn everything you can about what happened so you can do better next time and actually delight your customers. Over time you will probably become comfortable with rolling back at the first sign of trouble and just chalk it up to a learning experience.

Continuous Monitoring

As part of continuous deployment, you need continuous monitoring of your systems. You need to know what features people are using and how. From this you can learn where your users are running into problems and use this information to improve the situation for future builds.

You can also save yourself the headache of a Twitter storm if you detect a problem users are having before too many get mad about it and either fix or roll-back the annoyance.

The nice things about SaaS services is that you have complete control of the service and can track all sorts of usage metrics and information (of course subject to privacy policies).


A rather good blog post on WordPress’ experience with continuous deployment is here (the site this blog is hosted on). Just to emphasize that many companies are really doing this. In WordPress’ case deploying around 16 builds per day to customers. As companies perfect this process it is becoming a competitive weapon where whoever does this best, innovates faster and leaves their competition in the dust.



Written by smist08

February 4, 2012 at 1:45 pm

Automated Testing in Sage ERP Accpac Development

with 7 comments

All modern products rely heavily on automated testing to maintain quality during development. Accpac follows an Agile development methodology where development happens in short three week sprints where at the end of every sprint you want the product at a shippable level of quality. This doesn’t mean you would ship, that would depend on Product Management which determines the level of new features required, but it means that quality and/or bugs wouldn’t be a factor. When performing these short development sprints, the development team wants to know about problems as quickly as possible so any problems can be resolved quickly and a backlog of bugs doesn’t accumulate.

The goal is that as soon as a developer checks in changes, these are immediately built into the full product on a build server and a sequence of automated tests are run on that build to catch any introduced problems. There are other longer running automated tests that are run less frequently to also catch problems.

To perform the continuous builds and to run much of our automated tests we use “Hudson” ( Hudson is an extremely powerful and configurable system to continuously build the complete product. It knows all the project dependencies and knows what to build when things change.  Hudson builds on many other tools like Ant (similar to make or nmake) and Ivy among others to get things done ( The key thing is that it builds the complete Accpac system, not just a single module. Hudson also has the ability to track metrics so we get a graph of how the tests are running, performance metrics, lines of code metrics, etc. And this is updated on every build. Invaluable information for us to review.

The first level of automated testing are the “unit tests” ( These are tests that are included with the source code. They are short tests where each test should run in under 1 second. They are also self contained and don’t rely on the rest of the system being present. If other components are required, they are “mocked” with tools like easy-mock ( Mocking is a process of simulating the presence of other system components. One nice thing about “mocking” is that it makes it easy to introduce error conditions, since the mocking component can easily just return error codes. The unit tests are run as part of building each and every module. They provide a good level of confidence that a programmer hasn’t completely broken a module with the changes they are checking in to source control.

The next level of automated tests are longer running. When the Quality Assurance (QA) department first tests a new module, they write a series of test plans, the automation team takes these and transfers as many of these as possible to automated test scripts. We use Selenium (, which is a powerful scripting engine that drivers an Internet Browser simulating actual users. These tests are run over night to ensure everything is fine. We have a subset of these call the BVT (Build Validation Test) that runs against every build as a further smoke test to ensure things are ok.

All the tests so far are functional. They test whether the program is functioning properly. But further testing is required to ensure performance, scalability, reliability and multi-user are fine. We record the time taken for all the previous tests, so sometime they can detect performance problems, but they aren’t the main line of defense. We use the tool JMeter ( to test multi-user scalability. JMeter is a powerful tool that simulates any number of client’s accessing a server. This tool tests the server by generating SData HTTP requests from a number of workstations and bombarding the server with them. A very powerful tool. For straight performance we use VBA macros that access the Accpac Business Logic Layer to write large numbers of transaction or very large transactions, which are all timed to make sure performance is fine.

We still rely heavily on manual QA testers to devise new tests and to find unique ways to break the software, but once they develop the tests we look to automate them so they can be performed over and over without boring a QA tester to death. Automated testing is a vibrant area of computer science where new techniques are always being devised. For instance we are looking at incorporating “fuzz testing” ( into our automated test suites. This technique if often associated with security testing, but it is proving quite useful for generalized testing as well. Basically fuzz testing takes the more standard automated test suites and adds variation, either by knowing how to vary input to cause problems, or just running for a long time trying all possible combinations.

As we incorporate all these testing strategies into our SDLC (Software Development Life Cycle) we hope to make Accpac more reliable and to make each release more trouble free than the previous.

Written by smist08

April 5, 2010 at 5:24 pm