Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘Agile Development

The Sage 300 SDLC

leave a comment »

Introduction

A lot of my blog posts are to answer questions that I frequently receive. Then I have a ready answer of a blog posting link, or perhaps people read my blog and it saves me receiving an e-mail. This blog posting is along the same lines as I get asked quite frequently about our SDLC (Software Development Lifecycle). Usually this is in regards to someone trying to fill out a giant RFP full of questions that are mostly irrelevant to purchasing ERP software.

I covered various aspects of our development process in other blog posting which I’ll refer to here. Plus our process is always evolving as we learn from our experiences and try to improve. What I’m writing about here is specifically what the development team for Sage 300 ERP does, but a lot of it is also used by other Sage teams on other projects. There are always slight variations as the different teams do have their own preferences and as long as they follow the general standards that’s ok.

Within R&D we use the Agile Development Methodology, but R&D exists within a larger context within Sage, much of which doesn’t use Agile frameworks. As a result our Agile development has to fit somewhat within a larger non-Agile system that tracks and coordinates the various projects going on around Sage. This is to ensure all departments know what is going on and can plan accordingly.

Not-Agile

We have a general PMO department that tracks all the various projects. It coordinates getting approval for projects, determining release criteria and coordinating all the various departments such as Marketing, Product Management, IS, etc. So they can do their pieces at the appropriate time.

Initial product ideas come from our Innovation Process, but before converting an innovation idea into a larger development effort there is usually some POC work and some initial rough estimates that then lead to a project kickoff process where an initial business plan, along with an initial project plan are presented and either approved or rejected.

The project is then tracked by PMO as it goes through development and then at the end when the Agile part of the development is done, there is a release certification meeting where all the stakeholders get together that the solution is ready for customers. This includes that the software is ready and of a high quality, but also that support is ready, training material is available, back end systems are setup to take orders, marketing is ready and all the other pieces that go into a product launch.

Web_SDLC_GoodPMO

Also at this time we run a final regression to ensure that everything is working correctly. Generally this is only a couple of weeks as a sanity check that the Agile process below worked correctly.

Before making the product available to all customers, we first spend a few months in a controlled release with a handful of live customers to ensure everything works well for them. This not only tests the software, but their ability to be on-boarded, supported and trained. After this has proceeded successfully then the product is made available for everyone.

Some of these reasons for this non-Agile framework is that a number of parts of our organization having adopted Agile yet and eventually they will need to if we are to have truly effective cloud based products. The other reason that I blogged about here, is the need to coordinate many disparate teams working on different parts of a larger solution.

Agile

Within R&D we use the Agile Development Methodology. I’ve blogged about Agile development a number of times: here, here, here, here and here.

We’ve been using Agile programming for a number of years now. We use 2 week sprints, have sprint planning, maintain a backlog, have daily standups, sprint demos, sprint retrospectives and all the other aspects of Agile. We use VersionOne to manage our projects. In Agile you execute the backlog story by story and have very tight rules on what it means for a story to be “done”. This way as each story is completed, it is fully tested, documented and ready for use. The important thing here is to not build up a large list of defects that need to be fixed near the end of the project. Basically when the last story is finished (done) then the product should be ready to put in customer’s hands.

agile

The doneness criteria vary a bit by Agile team, but here are the doneness criteria for a team on one of our current projects:

  • All tasks items in a backlog story have a closed status
  • The code builds successfully without compiler errors and warnings and without ReSharper issues
  • The code is deployed to test environment successfully
  • The code is checked into the correct repository and branch
  • The code conforms to Sage Branding Guidelines.
  • The code performs to performance standards
  • The code is reviewed for applicable use of all documented standards.
  • The code is refactored and reviewed for good OOP implementation
  • The code conforms to UX wireframes, design and CSS guidelines.
  • Code coverage minimum percentage is 70% with a target of +90%
  • The UI displays correctly in all supported browsers (IE, Safari, Firefox, Chrome)
  • Unit tests are included and run successfully
  • Automation tests are included and run successfully in test environment
  • The environment has not been corrupted (database, etc.)
  • The QA and QA Review tasks are included and complete
  • Reviewed and accepted by the Product Owner
  • Implement and document any build and/or deployment changes
  • Knowledge Transfer Activities (wiki updated, code walkthrough, group code review, etc.)
  • Remaining hours for tasks are set to zero and the backlog item is closed

 

Ensuring stories are done is the key to a well-run Agile process, probably more important than a well-structured prioritized backlog.

As part of developing for the cloud, we want to release fairly regularly and can’t afford long manual regression tests. As a result we have a lot of emphasis on Unit Tests and Automated tests, such as those blogged here.

Similarly branching strategy, source code management, build management and continuous integration are also important parts of this process.

Summary

This was a quick overview of our Sage 300 SDLC. With any big project there are always a lot of moving parts and these have to be tracked accurately and reliably. Agile doesn’t mean that we don’t have deadlines or firm requirements. It does mean that we develop the most important things first and build quality in from the very start.

Advertisements

The Road to DevOps Part 1

with 5 comments

Introduction

Historically Development and Operations have been two separate departments that rarely talk to each other. Development produces a product using some sort of development methodology and when they’ve finished it, they release it and hand it off to Operations. Operations then installs the product, upgrades the users from the old version and then maintains the system, fixing hardware problems, logging software defect complaints and keeping backups. This worked fine when Development produced a release every 18 months or so, but doesn’t really work in the modern Web world where software can often be released hourly.

In the new world Development and Operations need to be combined into one team and one department. The goal being to remove any organizational or bureaucratic barriers between the two. It also recognizes that the goal of the company isn’t just to produce the software but to run it and to keep it available, up to date and healthy for all its customers. Operations has to provide feedback immediately to Development and Development has to quickly address any issues and provide updates quickly back to Operations.

In this posting I’m covering the aspect of DevOps concerned with frequently rolling out new versions to the cloud. In a future posting I’ll cover the aspects of DevOps concerned with the normal running of a cloud based service, like provisioning new users, monitoring the system, scaling the system and such.

Agile to DevOps

We have transitioned our software development methodology from Waterfall to Agile. A key idea of Agile is that you break work into small stories that you complete in a single short sprint (in our case two weeks). A key goal is that during the sprint each story is done completely including testing, documentation and anything else required. This then leads to the product being in a releasable state at the end of each sprint. In an ideal world you could then release (or deploy) the product at the end of each sprint.

This is certainly an improvement over Waterfall where the product is only releasable every year or 18 months or so. But generally with Agile this is the ideal, but not really what is quite achieved. Generally a release consists of the outcome of a number of sprints (usually around 10) followed by a short regression, held concurrently with a beta test followed by release.

So the question is how do we remove this extra overhead and release continuously (or much more frequently)? This is where DevOps comes in. It is a methodology that has been developed to extend Agile Development and principles straight through to actually encompassing the deployment and maintenance of the product. DevOps does require Development change the way it does things as it requires Operations to change. DevOps requires a much more team bases approach to doing things and requires organizational boundaries be removed. DevOps requires a razor focus on automating all manual processes in the build to deploy to monitor to get feedback cycle.

Development

Most development processes have an automated build system, usually built on something like Jenkins. The idea is that when you check in source code, the build system sees you checked it in, then it has rules that tell it what module it is part of, and rebuilds those modules, then it sees what modules depend on those modules and rebuilds those and so on. As part of the build process, unit tests are run and various automated tests are set off. The idea is that if anything goes wrong, it is very clear which check-in caused it and things can be quickly fixed and/or rolled back.

This is a good starting point but for effective DevOps it needs to be refined. Most modern source control systems support branching (most famously git). For DevOps it becomes even more crucial that the master branch of the product is always in releasable state and can be deployed at any time. The way this is achieved is by developing each feature as a separate branch. Then when a feature is completely ready for release it can be pulled into the master branch, which means it can be deployed at any time. Below is a diagram of how this process typically works:

Automated Testing

Obviously in this environment, it isn’t going to work if for every frequent release, you need to run a complete thorough manual test of the entire product. In an ideal world you have very complete test coverage using various levels of automated testing. I covered some aspects of this here. You still want to do some manual testing to ensure that things still feel right, but you don’t want to have to be repeating a lot of functional testing.

Operations

Operations can then take any release and in consultation with the various stakeholders release it to production. Operations is then in charge of this part of things and ensures the new versions in installed, data is converted and everything is running smoothly.

Some organizations release everything that is built which means their web site can be updated hourly. Github is a good example of this. But generally for ERP or CRM type software we want to be a bit more controlled. Generally there is a schedule of releases (perhaps one release every two weeks) and then a schedule of when things need to be done to be included in that release, which then controls which branches get pulled into the master branch. This is to ensure that there aren’t any disruptions to business customers. Again you can see that this process is blending elements of QA along with Operations which is why the DevOps team approach works so well.

A key idea that has emerged is the concept: “Infrastructure as Code”. In this world all Operations tasks are performed by code. This way things become much more repeatable and much more automated. It’s really this whole idea that you build your infrastructure by writing programs that then do the real work, that has largely led to movement of Developers into Operations.

DevOps

This is where Development and Operations must be working as a team. Operations has to let Developers know what tools (scripts) they require to deploy the new version. They need automated procedures to roll out the new version, convert data and such. They have to be working together very closely to develop all these scripts in the various tools like Jenkins, PowerShell, Maven, Ant, Chef, Puppet or Nexus.

Performing all this work takes a lot of effort. It has to be people’s full time job, or it just won’t get done properly. If people aren’t fully applied to this, manual processes will start to creep in, productivity and quality will suffer.

Beyond successfully deploying the software, this team has to handle things when they go wrong. They need to be able to rollback a version. Reverse the database schema changes and return to a known stable good state.

Summary

DevOps is a whole new profession. Combining many of the skills of Development with those of Operations. People with these skill sets will be in high demand as this is becoming an area that is making or breaking companies on the Web and in the Cloud. No one likes to have outages; no one likes to roll out bad upgrades. In today’s fast paced world, these can put huge pressures on a company. DevOps as a profession and set of operating procedures is a good way to alleviate this pressure while keep up to the fast pace.

On Adopting Agile Development

with 2 comments

We are slowly moving from a traditional waterfall software development process to a more agile one. We are looking to adopt scrum without going all the way to extreme programming (XP). As a first step we are looking to go to an iterative waterfall model. In this case we adopt agile iterations, but then do a mini-waterfall in each iteration. This would be a first step to learn some of the ideas of agile, rather than biting off too big a process change at once and risking chaos.

The waterfall method of software development has been around a long time, and many successful products have been developed using it. However it does have a number of drawbacks:

1. It relies heavily on all the requirements being defined upfront and then complete accurate specs being developed before coding begins. Practically speaking this has turned out to be humanly impossible. Requirements change as market conditions evolve. No one can review a large spec and ensure that something hasn’t been missed, or gone in a bad direction.

2. The handover from programming to QA leads to huge scheduling problems. You can schedule how long to code something (at least relatively) and how long to perform functional tests. But how do you schedule how many bugs will be found? How long to fix those? How long to retest  these? How long QA is delayed because they are blocked by these bugs?

3. Combining 1 and 2, practically speaking the worst design defects are found in QA when it is quite expensive to fix these. Again this tends to be un-sheduleable work and usually is the cause of the biggest product delays.

Agile looks to fix these problems (and many more besides). The idea is to keep units of work manageable without requiring super-human powers. Another idea is that you don’t need the requirements up front, not do you need the specs up front (if at all). Some of the key ideas:

1. Break down requirements into small easy to manager “user stories”. This makes the requirements more customer focused and really forces functional decomposition right from the beginning. It then lets you prioritize these requirements and tends to limit designer gold plating.

2. All team members are involved for the whole project. There are no hand overs from one group to another. Traditionally these hand overs lead to delays (due to people not being available at the right time), defects due to misunderstandings, and lack of teamwork. With agile the design analyst, programmer, ucd specialist, QA person and product owner are always working together as a team through the whole process.

A lot of the rest of the Agile Development Process is then executing these ideas in a best practices type environment using the experience of previous teams. From the experience of others, leads to the scrum principles:

1. Short fixed length iterations (usually 2 or 3 weeks).

2. The product is “shippable” quality at the end of every iteration.

3. How to break down the user requirements,  and pick the ones to implement in each iteration.

4. How to prioritize the requirements and handle things that go wrong, like too hard user stories, key team members on vacation, etc.

Generally many product groups have discovered that Agile Development Process’s are more productive (more features or products are developed), quality is higher and the process is more predictable. Probably the hardest thing with adopting Agile is just to forget all those practices taught in older Software Engineering courses in University and accept the newer methods.

Written by smist08

April 4, 2009 at 4:42 am