Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘mas

Test Driven Development at Sage

with 7 comments

As the Sage ERP development teams continue their journey into Agile Development, we are looking to even more fundamentally change the way we do things. We have adopted the Scrum methodology and are now operating in two week sprints. Within those sprints we take user stories and complete them to very exacting “doneness” criteria. The goal is to have a releasable product at the end of each sprint (releasable in a quality context, but not necessarily functionally complete). So how do we perform all the development, QA, UCD and documentation within small two week sprints? What we are looking to do is to truly use a Test Driven Development Process. This blog post is intended to outline what some of our Agile Teams are already doing and what the rest are looking to adopt soon.

We switched from using the Waterfall Software Development Methodology to the Agile Methodology which gave us a number of good gains. The main one being containing bugs to sprints, so we don’t end up at the end of a project having to fix a huge number of bugs in a QA or regression phase after code complete. However there is a lot of temptation for people that are used to the Waterfall Methodology to start doing mini-waterfalls with-in the Agile sprints. Then for each story, they would design – code – QA –bugfix the feature as a little waterfall within the Sprint. In Agile the QA and programmer should work together and there should be no separation of work like this. As a first step to stopping this we shortened our Sprints from three weeks to two weeks. This made it very hard for people to do mini-waterfalls within the sprint since there just isn’t enough time.

Another goal of Agile Development is to make extensive use of automated testing so that rather than having to continually manually test the product to ensure bugs aren’t creeping in, instead there are automated tests that run on every build of the product to ensure this doesn’t happen. There can be a perception that adding automated tests takes too much time and slows things down. This doesn’t take into account that finding and fixing bugs takes way more time and resources.

To solve these problems we have adopted Test Driven Development within the Agile Sprints. Here we reverse the development process to some degree and write the tests first. This has a number of benefits:

  • QA team members don’t need to wait for programmers to finish coding before they have anything to do.
  • Automated tests are developed as part of the process and are fundamental to the process.
  • This forces the story to be well defined up front since the first thing you need to do is define how to test it.

Immediately the biggest gain we got from this approach was far better defined stories with far better acceptance criteria. This saved a lot of re-work where programmers implemented something without fully understanding the story or acceptance criteria and then the story would fail because of this.

The diagram below shows the steps we follow executing stories during an Agile Sprint (developed by the Accpac Super team):

Notice how we write the test cases first as we investigate the story. Only then, after this is all understood and reviewed do we actually do the implementation.

A common problem with Agile is that programmers will code quite a few stories and then give them to QA near the end of the sprint. This overwhelms the QA testers and doesn’t give them sufficient time to do all their testing. With this new approach, all the test cases are written up front and well defined, then if too many things end up at the end of the sprint, other people like Business Analysts, Writers or Programmers can help execute the test procedures to complete the testing.

So far this sounds fairly manual. But there are good tools to help automate this process.

Fit and FitNesse

FitNesse is a Wiki and an automated testing tool. It is oriented around doing acceptance testing rather than unit testing. A lot of times programmers look at this tool and dismiss it as another unit testing framework. The difference is subtle but important. For the programmer Fit is similar to unit testing frameworks, the work the programmer does is very similar to writing unit tests. However you are more writing an API for writing tests than the actual tests. Then FitNesse gives a mechanism for QA (or other non-programmers) to feed input into these tests to cover all the necessary test cases. In the Wiki you document the acceptance criteria and enter all the tests to prove that the acceptance criteria are met. Then since these are automated tests they will be run as often as desired (perhaps on every build) to ensure this story’s acceptance criteria continue to be met.

We use EasyMock to be able to test components in isolation, so modules can be tested without requiring the whole system be present. This makes setting up the tests far easier, makes it very easy for them to run in the build system environment and allows them to run very quickly (since things like database access are mocked).

We’ve found this to be a very powerful tool to document and produce automated tests. It allows QAs who know the product and what it should do, to create automated tests without having to be programmers. You can probably find standalone tools that do the individual parts of FitNesse better, but the power comes from how they work together and enable the team to create the tests and automate the acceptance testing.

Jenkins

Jenkins (previously known as Hudson) is a continuous integration and build tool. It uses Ivy to define all the dependencies between all the modules in a system. It then scans source control (we use SubVersion) to see when files change. When it detects that a file has changed it will re-build any modules that it is part of and then based on the Ivy dependency map, it will build any modules that depend on these and so on. As part of this process it can run any associated automated tests, run tools like FindBugs or Emma as well.

The goal of this then is to report back to a team as soon as possible that something they have done has broken something else. Bugs are cheaper to fix the sooner they are found. If they are found right away while some one is working on the offending code, they can usually be immediately and easily fixed. Combining Jenkins and FitNesse is a great way to accomplish this.

Regression Testing

In addition to unit testing and automated acceptance testing, we also run full automated tests on the whole system.

The Accpac team uses Selenium. Selenium drives the system by driving the Browser. It uses the automation interface to the Browser to drive our system. This then looks to the system like an end user and all parts of the system are used. I blogged previously about our process of moving the VB screens to the Web, here I described our Porting tool to initially move the VB screens to SWT. When we generate the SWT project, we also generate a number of Selenium tests for each screen automatically. So as a starting point we have automated Selenium tests to drive the basic CRUD functionality of each screen.

For non-Web products like MAS, Peachtree, Simply Accounting and Peachtree, we use SilkTest for automated system testing. We have developed fairly extensive suites to tests to provide good automated coverage for all these products.

To the Cloud

What we are really striving to do is that after each Agile Sprint we are in releasable state. This means that we could post the result of each Agile Sprint to a hosted site where real customers are running our software. These customers would get the benefits of whatever improvements we made during the Sprint with no worry that we have broken anything.

Whether we actually ever do this is a separate question, but we are striving towards having our software always in a releasable state, so we can make this decision based on the business and market needs. The trend in the software industry is towards frequent smaller releases rather than giant infrequent releases. Generally this leads to a much faster pace of innovation (just look at how fast Google Chrome is evolving) and provides much faster customer feedback into the development system resulting in software that matches customer needs far better.

Although most of our competitors aren’t operating like this yet, eventually they will or they will be replaced by companies that do. We want to ensure that Sage can develop software as efficiently as anyone and that we have top product quality. We also want to ensure we are competitive and can innovate at a very competitive pace. Hopefully you will see this over the coming product releases.

Summary

We are still learning about effective test driven development at Sage, but already the teams that are pioneering its use are seeing great results. We’ve done automated regression testing with tools like Selenium and SilkTest for some time now and seen good results, now it’s a matter of taking it to the next level. The goal being to shorten our development/release cycles while increasing product quality at the same time.

Written by smist08

April 16, 2011 at 5:31 pm

SaaSifying Sage

with 9 comments

Sage is a large company with many products, mostly ERP and CRM for various sized businesses. Sage has many on-premise products such as Sage ERP MAS 90, Sage ERP MAS 500 and Sage ERP X3. Sage has many SaaS cloud/web based products including SageOne, Sage Spark and Pastel MyBusiness Online. Sage has many products that are sold and deployed both ways such as Sage ERP Accpac and Sage SalesLogix.

SaaS strictly speaking stands for Software as a Service and usually refers to companies that host web sites that perform various applications such as NetSuite and Salesforce. These companies try to sell against on-premise vendors siting the following advantages:

  • No upgrade costs, upgrades are transparent and continuous (and no extra cost).
  • Easy per month subscription pricing (no up-front large purchase)
  • Customers don’t need to maintain any infrastructure such as servers
  • You can access their applications from anywhere from any device

On-premise vendors then answer with the following advantages:

  • You own your own data. (SaaS vendors have had customer data auctioned off when they’ve gone out of business).
  • You control data security; your data is never on the Internet.
  • You control when you upgrade, it’s never done unexpectedly and so can’t disrupt your business.
  • You have your own copy of the program that you can customize as much as you like.
  • The cost of direct purchase of the software over the life of the software is cheaper than subscription pricing.

Neither side is perfect at any of these. But it gives an idea of how the debate goes. From a development point of view, SaaS has the appeal of only supporting one version. Right now at Sage we support the current version along with two prior versions. This support takes R&D resources investigating problems and providing updates. There is a lot of appeal to R&D concentrating resources on the current version and ignoring anything that came before (don’t worry I’m not going to be allowed to adopt a one version support policy).

Cultural Change

Here at Sage we want to try to bridge the gap and try to provide products that are the best of both worlds. Most people think that you need to choose the advantages of SaaS OR choose the advantages of on-premise. We want to change that OR into an AND, providing the advantages of both combined. Doing this will be difficult, but technologies and processes are evolving and advancing to a state where we can start to do this in many areas.

But the big change is cultural. We are changing our corporate culture from being an on-premise vendor to being a SaaS vendor. Even for products that will never be SaaS, that will always be on-premise, we will develop them in the same way that SaaS vendors develop products. In this way we will pick up many of the benefits of SaaS without losing the benefits of on-premise.

At the same time we will be developing and releasing many of our products both as SaaS or cloud based solutions and on-premise.

Agile Development Process

As a starting point all R&D groups at Sage have either already adopted or are in the process of adopting the agile development methodology. This is the methodology used by most SaaS vendors to allow them to continuously update their applications. The concept is to develop in small “sprints” which are typically two or three weeks long. Everything started in a sprint must be finished in a sprint. The goal is that at the end of each sprint, the product is ready for release. A good SaaS vendor can actually deploy every sprint end build to their live site. One of the keys to accomplish this is to have a very large and complete set of automated tests that can validate that each build works properly and has very high quality. Accpac has used Agile for about a year to create the 6.0 release. MAS has used it for two years for their 4.45 and 7.3 releases.

With Agile development we don’t accumulate quality debt like the waterfall method, so we don’t have long Q.A. and regression test phases. This makes our development more efficient. Also creating all this automated testing causes bugs to be found much quicker making fixing them much cheaper. Another side effect of all this automated testing has been to improve the general quality of our releases.

Part of Agile is that you have a Product Backlog that consists of a list of everything you want to do with a product. New feature ideas are added here. Then a set of Product Owners order this list by priority and development always takes the item at the top of the list to do next. This way R&D is always working on the most important thing and then if we do more frequent releases with fewer features then there should still be a lot of value since we are doing the most important things for the customer. This also means that for a customer, that if something doesn’t make the cut, they don’t have to wait for a future 18 month later release, then only need to wait for a future 3 or 4 month release.

Ease of Installation

Ideally a SaaS vendor will update their product on a regular basis, perhaps every few months. In an ideal world the only thing the customer notices is suddenly some great new features appear. Otherwise the customer didn’t have to do anything. Practically speaking, many SaaS vendors tend to regularly break customers with features and cause training problems, remove key features or cause bugs. However if a SaaS vendor does this badly they will be replaced by a vendor that does this well in short order.

In the mid-market on-premise world, vendors have the bad habit of relying on their business partners. They (we) put in a great new feature, and it makes upgrading hard. Now we have a choice, we can spend the time to make the upgrade transparent or we can add another new feature. There is far too much temptation to add the new feature and leave it to the Business Partners to deal with the upgrade difficulties. Then we are surprised to hear from customers complaints about large upgrade costs.

As we go to more frequent releases for our on-premise products, we can’t keep this up. We are now spending more resources making installation and upgrade an easier process. For instance a main goal of the new MAS-90 business framework is to allow customization via object oriented programming rather than source code. Moving source code customizations from version to version is hugely expensive and as a result, we are phasing out source code customization in all our products. Similarly Accpac combined our install and data upgrade process into one per release from one per module per release to ease upgrading. Further we are working to make all report and screen customizations completely safe from version to version.

We still have a long way to go, but ultimately we would like our on-premise products to seamlessly download new versions from the Internet, update themselves and continue working without any extra work being required. We would of course allow customers and business partners to control this process, but we would ensure all data is migrated seamlessly along with all customizations. But the goal is to make upgrades as seamless and painless on-premise as SaaS.

Deployment

With modern virtualization technologies we can offer any of our on-premise applications hosted in our data center (i.e. in the cloud). These are then accessed with Citrix client or remote desktop and the client can use these just like they are on-premise but without the expense of buying and maintaining the servers. These are also offered via subscription pricing and common maintenance tasks like database backup are taken care of. With this model customers can move back and forth between cloud and on-premise as they wish, or have some locations access via the cloud and some run on-premise. We also facilitate that customers can take a copy of their databases at any time so they can be re-assured they won’t lose anything. You can even access your application from an iPad using the Citrix iPad application. This option alone gives many of the advantages of both SaaS and on-premise combined.

Then many of our applications are moving to Web interfaces such as Sage ERP Accpac 6. These will also be available via SaaS and on-premise, but the SaaS version will be a true multi-tenant SaaS version similar to our competitors. Here our applications will have true HTML/JavaScript based interfaces that run in any Browser on any device, so you gain all the advantages of SaaS and have many of the options as on-premise.

Customization

Today on-premise applications have a huge advantage in customization capabilities over SaaS products. However technology is changing and demands are changing. Many SaaS vendors started with no customization capabilities and have been bolting on customization features ever since. However technology is improving and the ability to customize SaaS is greatly increasing. Plus the trend is for customers to be shying away from such heavy customizations due to the large development and maintenance costs.

All Sage products are built with a customization model to start with. We don’t have to bolt a customization model onto a fixed schema database model (where all customers share the same database schema). So as we move our products to SaaS, all the customization capabilities come with them.

The one change is that for SaaS we have to move away from true source code customization. Where you just modify any source code you like in the system. You can certainly write code and extend our system in an object oriented manner, but you can’t change the underlying objects that make up the core functionality. This is a more disciplined approach, but it allows customization in the cloud and it keeps the costs down for customers, since this allows these source code customizations to be upgrade safe.

Summary

Sage is changing to do all software development like an ideal SaaS vendor. We are looking to bridge the gap between on-premise and SaaS based applications. We are pursuing the ambitious goal of combining the best of both worlds.

In much of Sage’s marketing material you see the three columns that we are pursuing through development:

To a large degree this involves improving our current on-premise products, including offering flexible deployment and subscription pricing models and simplifying the upgrade process. Then developing SaaS versions of our key strategic products including full modern Web Based interfaces and then connecting the two worlds with our connected services strategy.

Written by smist08

February 20, 2011 at 8:07 pm