Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘scrum

Doing Work in Small Batches and Limiting Work in Progress

with 5 comments

Recently Software Development has been borrowing a lot of ideas from Japanese Lean Manufacturing. In a way this mirrors the progression from the Detroit way of building cars pioneered by Henry Ford to the modern way pioneered by Toyota.

The original Ford production line approach was that a car was assembled on an assembly line by going through a series of specialized stations that were optimized to do one things really well and efficiently, so the car would go from perhaps the attach doors stations to the add Window glass station. Each station was optimized to do its job really well and the priority was to keep the assembly line moving no matter what. If the assembly line ever stopped then the hundreds of people working on it would be idled and this was considered a disaster. If problems occurred then they were noted and fixed post-assembly line. This process was then improved by optimizing each station to be quicker and more automated. Generally this produced a very good system of producing a very high volume of identical cars.

From the outside a Toyota assembly line looks nearly identical. But it operates very differently. Toyota didn’t have the same large market as Detroit and didn’t need the volume. They needed to produce more specialized cars and needed to switch their assembly lines quickly from one run to another. Their emphasis wasn’t on making very specialized machines to do one task quickly; it was on producing general machines that could do a number of different things and could be switched quickly from doing one job to another. Toyota then does small runs on their assembly line rather than one massive long run. The big benefit to doing things in small batches was that quality problems are found much sooner. Hence they can stop the assembly line on the first car and fix a problem, rather than producing thousands of cars with the problem and then possibly not fixing the problem because it’s too expensive at that point. This also re-enforces the practice of fixing problems right away rather than letting them pile up and that it’s more cost effective to have the production line idle while the problem is fixed than having to fix it later.

Another example from the book “Lean Thinking” by James Womack and Daniel Jones recounts a story of one of the authors stuffing envelopes with their two young children. The children wanted to take the approach of folding all the letters, then putting all the folded letters in the envelopes then sealing all the envelopes then putting stamps on all the envelopes. This seems like an efficient and intuitive way to approach the problem. However the author insisted they do it the less intuitive way of one complete envelope at a time, i.e. folding one letter, stuffing it in an envelope, sealing it and putting a stamp on and then going on to the next one. Which way is faster? It turns out that researchers have timed groups of people performing this task and it turns out the one at a time approach is faster! It has a couple of other advantage as well. One is that if there is a problem, you find out right away; if the envelope is the wrong size, you find out on the first letter, not after you have folded all the letters, this could save you then having to refold all the letters to fit the envelopes. Similarly if you take a break you know how far along you are, if you’ve completed 40 out of 100 letters then you are 40% done and you can estimate your remaining time easily. You can’t do this when you do each step completely. This process is called “single piece flow” and is an example of doing a task as lots of small batches rather than one large batch. This is a general principle you want to apply in everything you do, resisting the old Ford type assembly line type mentality. You also receive the first finished product off the assembly line quicker, so you can validate it with your customer before too many more are produced to ensure you are producing the right thing.

Kanban vs Scrum

To some degree batch size is at the heart of the difference between the Kanban and Scrum methods of developing software. Scrum tends to batch things into sprints (usually two or three weeks long) which is much smaller than traditional waterfall development, but Kanban wants to take it further. In Kanban you only work on one or two small things at a time and finish one before starting the next things. You operate on lots of small batches, but you also have to limit how many batches (or tasks) you can have in progress at a time. This is called limiting work in progress (WIP). In Kanban you don’t worry about batching things up into sprints, just keeping the jobs small and limiting WIP. Kanban requires a lot of discipline and often if a development team hasn’t successfully progressed from waterfall to scrum, then adopting Kanban is a disaster.

The goal here is to produce small high quality increments that are immediately validated by customers. This way if you are going in the wrong direction, you won’t get too far before discovering it and not too much time will have been wasted (only a couple of small batches).

How do you get customer feedback, one small increment at a time? Obviously you can’t have dozens of customers download, install and play with a new version of your product every time you add a small increment and then interview them about it. But there are techniques you can use to do just this. The first is to produce your software via “continuous deployment”. This means that every build of your software is automatically deployed to customers whether that means pushing it to a web server or loading it on an automatic software updater service. You need to be able to control which customers get it and you need to be able to roll it back if something goes wrong. To be confident with this sort of procedure you need really good automated testing in place to ensure you haven’t accidentally broken something. Then to find out whether your increments (or batches) provide customer value, you need really good instrumentation in your product to report back on customer usage, whether they use the feature, how long it takes to perform tasks and any other useful statistics.

Software products are large and complicated systems. Managing that complexity is a major challenge. Breaking large tasks down into many small tasks is a powerful tool to reducing complexity. Combining many small but high quality parts results in emergent complexity, but again it is much easier to manage by adding one small thing at a time rather than suddenly throwing several large items into the mix at once.

Summary

Most of the ideas in this blog posting come from my reading the “Lean Startup” by Eric Ries over the holidays. Quite a good book with many useful ideas on product development and innovation. Reducing Batch size and limiting WIP appeal to me to help make Sage more agile and innovative as we move forward.

Written by smist08

January 8, 2012 at 12:35 am

Releasable State

with 5 comments

First an advertisement for a new session just added for Sage Summit:

Session: Sage ERP Technology Roadmap on Sunday, July 10 at 1pm – 2:30pm.

Session Description: Join Sage ERP Product leaders who will be discussing the technology evolution of Sage ERP product lines. This session will cover our product journey to the cloud; technologies being used to deliver rich web experience; developments on the Connected Services front(joining On-premise applications with the Cloud enabling us for web and mobile services); what kind of tools/technology/skills are needed to integrate, customize our products in the web world; and collaboration occurring on common components. There will also be time set aside for open dialogue. This session is for Sage ERP Development Partners only.

Now back to our regularly scheduled programming…

Releasable State

One of the key goals of Agile Software Development is keeping software in a releasable state. There are quite a few reasons that this is a good thing which we are going to explore in this blog posting. This is very much related to test driven development which I blogged about here.

Problems with Waterfall

In traditional Waterfall Software Development you have a number of key phases namely the setting requirements phase then the design phase then the implementation (or coding) phase then the QA phase then the maintenance phase. The diagram below gives an indication why this is called Waterfall.

There are a number of problems with this; one of the key ones is that bugs and defects are found quite late and that these tend to accumulate to quite large numbers. There are other problems with the requirements gathering and design phases but those would be a topic for another day. Generally it’s far cheaper to fix bugs right when they occur and everything about the code is fresh in people’s minds. As the time between coding and discovering a bug increases, the cost (or time) to fix it increases quite quickly. Fixing a bug from two or more stages in the development process ago is especially expensive. These bugs also cause havoc to the scheduling process. Estimating how long it takes to code something is usually marginally accurate. Estimating how long it will take to fix say a hundred or more bugs is really a crap shoot. Usually when Waterfall projects start to fall behind, there is a feeling that you can make it up by squeezing the verification phase, but practically, what happens is that as a project falls behind then the verification phase grows faster, since more bugs are found and they take longer than expected to fix.

Besides making scheduling un-predictable, what often happens in this environment is that you miss a couple of deadlines; the product is feature complete and has a couple of successful beta sites. But QA are still finding quite a high number of bugs and the outstanding bug count is high. What tends to happen next is a rationalization that the bugs that QA are finding won’t affect customers or that they are things like adherence to standards or in obscure corners of the product and that given the two beta sites are happy then why not go ahead and release. The product is then released; many bugs are found in the field, customers are very unhappy. Then there is a huge panic to address these bugs in a first service pack. This then delays the start of development of the next version and guarantees the process will repeat itself. This then lead to quite a dangerous downward spiral.

Generally this is a gloomy prospect that leads to the various software development death marches that you hear about.

Agile to the Rescue

Agile Software Development wants to address this problem in a couple of ways:

  1. Fix bugs right away, don’t let the time between coding and bug fixing happen.
  2. Don’t let bugs accumulate. Fix bugs right away, it is never ok to let bugs accumulate for later.

One of the main tenants of Agile Development is that you want to keep the software in a releasable state. You never want to get yourself into a situation where you know you have to fix a lot of bugs before releasing. The philosophy has to be that if you know about something in the product that would prevent it from being released right now, then that must be addressed immediately. This statement refers to quality and not feature completeness. Features are added one at a time off a priority queue and then it’s up to Product Management to decide the commercial readiness of the product.

Sure you could try to apply these two points to Waterfall, but the process and philosophy of Waterfall works against you. The goal to finish each phase on time tends to drive away good intentions to not let this happen yet again.

The Agile Development process operates by building a product backlog of user stories that you want to implement into the product. These are in a priority order (by business value) so the teams take stories from the top of this queue to implement. These are then implemented in two or three week “sprints”. At the end of each sprint you produce a working increment of your product. This is shown in the diagram below:

The stories in the product backlog are small enough that they can be fully implemented, tested and documented inside of a single sprint. When done, stories need to be “accepted”. Stories are not done or accepted unless fully tested, code reviewed, documented and complete. If a story is not accepted then it isn’t made part of the product, the team doesn’t get any credit for the work and it will need to be completed next sprint. The key point is that at the end of each sprint there isn’t a bunch of half-finished work checked in causing bugs and general havoc. From the diagram you might want to think of each sprint as a mini-Waterfall, however that would be wrong, a sprint is quite different, but again that’s another topic, perhaps for a future post.

If you are a SaaS product, potentially you could deploy the output of each sprint to your live web server and have all your customers immediately benefit from the work performed last sprint. Within Sage we are trying to operate with a SaaS mindset and I previously blogged about that here. Once you have working software, you have a lot of options as to what to do with it. You have far more opportunities to distribute your software, if not live, then to get more customer feedback to provide more improvements in later sprints.

A key to staying in a Releasable State is having a good set of automated testing tools that can be run continuously on the product so that bugs are found immediately and not allowed to creep in. I blogged on our efforts to become a more test driven organization here.

Generally with-in Agile you are trying to reduce accumulated in quality debt and technical debt. These are defects or parts of the program that badly need updating. The more of these that you accumulate, the harder a program is to maintain. With-in Agile you track these and work very hard to ensure you are always reducing these.

The key thing missing from the diagram above on Agile Development is the lack of any sort of regression or final quality validation phase. This is the desired end state; it takes a bit of time to get to a place when you are really confident you can drop these. We are endeavoring to shorten them with each release with the goal of removing them entirely one day. This speeds up our releases, by eliminating this work and makes things more predictable, since otherwise going into regression you don’t know what you will find, and then the time to fix anything found is also unpredictable.

The further you let a project get from releasable state, the less predictable the completion date. Worse you don’t know how much of this accumulated quality debt you are going to have to pay to release, how negatively this will affect customers or how badly this will hamper future product maintenance. To work optimially you need to really keep a hard line on these. The idea of being able to release at the end of every sprint is a good rallying cry to keep these all under control.

Summary

The Agile Development Process’s goal of fixing bugs right away as they are introduced, performing continuous automated testing and removing the regression testing tail end processes provided software development organizations far better predictability in their development efforts.

This also makes your development process far more adaptable in changing direction as business and competitive conditions change. Since it is far easier to change direction when going from releasable state to releasable state as you go from short sprint to sprint.

Written by smist08

May 28, 2011 at 4:17 pm

On Agile Sprint Reviews

with one comment

On part of the agile scrum development process I like are the regular sprint reviews. Basically in Scrum, you operate on a really short mini-release cycle called a sprint. Our sprints are 3 weeks long and on the last day of the sprint we hold a sprint review, which is a presentation on what was accomplished during the sprint. We have seven agile teams which all operate on the same sprint calendar. At each sprint review meeting, each team gets 15 minutes to present what they accomplished in the sprint. Attendance of this meeting tends to be quite high and diverse including all the sprint team members, people from Product Management, Support, Sales, etc. It is a really good way for everyone in the company to get a view to how development of the next release is going and provide feedback.

The best demos in the sprint review are of running the real product and accomplishing real user tasks (or stories). But sometime the demos can be of low-level processing as seen from the debugger or SQL analyzer, user centered designs, test cases, etc. The sprint review is a great motivator for the teams to finish stories and to be able to present them. There is a lot of competition between the teams to show the most progress and to present the most amazing demos.

The sprint review is just one small aspect of Agile programming, but it acts as a true focusing force to bring the product back together at the end of each sprint and to motivate the teams to finish their stories in time for the sprint review. The couple of days before the sprint review can be quite stressful for the teams as they try to finish things up. Remember Agile development has quite a strong “doneness” criteria which must be met before something can be demo’ed in the review. Any story that is demo’ed must be “done”, that means it must be fully finished, meaning fully tested and accepted by the product owner. If the story isn’t “done”, then it isn’t finished and can’t be demo’ed. It must be left to the next sprint and the team won’t get credit for that story until it meets the full “doneness” criteria.

Written by smist08

November 8, 2009 at 3:53 am

Learning About Agile

with one comment

For Sage Accpac ERP development we adopted the Agile Scrum development methodology back in June. Since then we are now on Sprint 5 and learning all sorts of things about scrum and agile as we roll out and practice this methodology. There are quite a few aspects to agile, but one of the key ideas is to relentlessly breakdown large tasks into smaller manageable tasks. Then to finish those tasks completely, don’t let them drag on, say code now and leave it to be tested three months later. For us we’ve realized quite a few immediate benefits:

  1. Teamwork. Our agile teams are typically 6 to 12 people (most close to 6). This fosters excellent teamwork and shared purpose for the team members. It is also far easier to manage such a team, than say manage 40 people working on the same thing. The teams are also cross functional, so it aligns programming, quality assurance, user centered design and documentation together on the same team, rather than being members of separate departments with their own agendas.
  2. Sprints. By running in three week sprints, it forces us to keep the product integrated and greatly helps preventing large numbers of bugs accumulating that need to be cleaned up later.
  3. Better estimation. By forcing us to break tasks down to take less than one sprint, it forces us to fully decompose tasks and greatly increases our estimating accuracy in the process.
  4. Better release planning. The agile release planning process is far superior to the older waterfall methods, with all their hand overs. With agile release planning you create your story backlog and when planned this is what you directly act on and track, without an conversion processes.
  5. Better project tracking. Tracking agile stories and story points and measuring team velocities is far easier than traditional project management techniques which track by days or hours with all their complicated formulas.
  6. Pushing decision making down to the teams. This has greatly eliminated roadblocks and enhanced teamwork. The teams make their own decisions and are responsible and accountable for the results. Works far better than relying on a Directory or Manager for decision making.

When adopting any new process there is some pain as you switch from what you know and what is tried and trusted to something more unknown. But it’s really a matter of having the patience to give the new process a chance. To see it working and to learn to trust it. One of the harder things to accept for Agile is the concept of not designing things until they are needed. There is an element of trust and confidence here that you will be able to successfully do this when the time comes. That everything isn’t laid out at the beginning of the project.

Written by smist08

September 20, 2009 at 12:34 am