Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘kanban

On Organizing Agile

with 6 comments

Introduction

Generally Agile Programming is first taught as a very simple process, only involving some simple tools found in most kindergartens. The basic ideas of Agile Programming are quite simple and designed to make software development easier with less wasted work. However many people that adopt Agile find this transformed into quite a rigid bureaucratic system that isn’t very productive. What are a few things that can go wrong? How can you return to the simplicity of the original Agile goals? How can you track and manage large projects with many teams and dependencies in an Agile world?

Agile Manifesto

Agile was started by a group of programmers who came up with a simple set of guiding principles.

Mini-manifesto1

The purpose of this wasn’t to obliterate all the items on the right, it was to set some priorities. I think a lot of times when organizations adopt Agile, they follow this initially, but over time develop fixed ways of doing things and begin to have rationalizations that when they are doing the things on the right, they are really doing the things on the left. It takes a bit of organizational self-awareness to stay on guard against this.

Organizing Big Projects

A lot of simple courses on Agile focus on projects executed by one sprint team. Then to scale the process they wave their hands and just say you have multiple sprint teams executing off the backlog. Sounds good but tends to neglect a few factors. One is that different teams develop different expertise. Perhaps because they have a lot of experience in a given area or they have been tasked with specializing in a specific area like perhaps interaction design. Often in a large project it is very inefficient to have all the teams taking items from the same backlog. Often it’s better to have the teams specialized on separate parts of the project. People want to be masters of what they do, and trying to make everyone a jack of all trades just makes people unhappy. Further software is complex and subtle, having teams really know a specific area can be a huge help to move things ahead.

Once you start developing these specialization, you start introducing dependencies where one team will be waiting for another. Once you have dozens of teams, then you will need some sort of Project Management structure in place to track of this. A simple burn down of a backlog just doesn’t cut it.

Agile versus Waterfall

I find that the Agile versus Waterfall debate tends to indicate problems. People will say they won’t do something because it’s “Waterfall” and not “Agile”. But the truth is that the elements of Waterfall still need to be done. There has to be some program design, there has to be some architectural design, and there has to be some documentation. There is going to be integration and system testing. Just the goal is that these aren’t the main activities and they are generally done in much smaller increments than Waterfall.

When people refuse to do tasks because they are “Waterfall”, it is usually a warning sign that something important is going to be missed because someone dislikes a particular task. Generally your ears should perk up when you hear this argument being used.

Are You Really Agile?

Another problem area I see is when Agile teams become very non-agile. By this I mean they do sprint planning and then for the duration of the sprint absolutely refuse to do anything that wasn’t specified in sprint planning.

This can be good for getting the backlog reduced. But if priorities change in the backlog, it could mean they are now doing the wrong things. This usually means that discoveries and knowledge learned in a sprint isn’t used until the next sprint. Further if a customer issue comes up, it can look very bad to customers if their problem isn’t going to be even considered till next sprint planning.

Further if your product runs in the cloud, you need to be instantly responsive. In this case your teams do need to drop everything when your live system has a problem. You can’t let your cloud service be down until the next sprint.

Kanban

One commonly proposed solution to the above is to use Kanban. Here you don’t do sprint planning and always execute the top priority items in the backlog. There is further discipline to help make this work like limiting work in progress. To have Kanban work once you have a production system requires really strong product owner to avoid having all capacity going to the production system. Often this means holding a hard line on what is really important for customers, fixing minor problems or introducing more important new features.

Working with Off-Shore Teams

If you are working with off-shore teams then documentation and communications become much more important. You also need to be careful not to introduce bottlenecks that would stop them getting their work done (like waiting for a remote product owner to update/input backlog stories).

droppedImage

Most big multi-team projects usually have at least one off-shore team these days. In order to scale product development departments being able to work effectively across multiple locations in very different time zones becomes crucial.

Often if you are having problems with your Agile development process, then adding an off-shore component will magnify them and lead to much more serious problems.

Similarly teams are often not so co-located due to team members working at home, perhaps in a different city. There are many new technologies to help this work as I blogged on here.

Backlog Priorities

Setting backlog priorities is often quite controversial because of the mixed needs of different members of the product owner community. There has to be a balance of what is needed for customer, what is needed by dependent teams as well as architectural and operational issues. It takes a strong product owner with a clear understanding of these issues to set the priorities appropriately. Further the product owner needs the power to stick to his priorities and not to be continuously overridden by people escalating issues up the chain of command.

Summary

Keeping large projects Agile is a real challenge. Keeping Agile teams Agile over the years is a major challenge. People like routine and are always trying to get a nice regular fixed routine. This can be nice, but not entirely realistic as everyone tries to do things in Internet time while handling problems in cloud services in real time.

Advertisements

Doing Work in Small Batches and Limiting Work in Progress

with 5 comments

Recently Software Development has been borrowing a lot of ideas from Japanese Lean Manufacturing. In a way this mirrors the progression from the Detroit way of building cars pioneered by Henry Ford to the modern way pioneered by Toyota.

The original Ford production line approach was that a car was assembled on an assembly line by going through a series of specialized stations that were optimized to do one things really well and efficiently, so the car would go from perhaps the attach doors stations to the add Window glass station. Each station was optimized to do its job really well and the priority was to keep the assembly line moving no matter what. If the assembly line ever stopped then the hundreds of people working on it would be idled and this was considered a disaster. If problems occurred then they were noted and fixed post-assembly line. This process was then improved by optimizing each station to be quicker and more automated. Generally this produced a very good system of producing a very high volume of identical cars.

From the outside a Toyota assembly line looks nearly identical. But it operates very differently. Toyota didn’t have the same large market as Detroit and didn’t need the volume. They needed to produce more specialized cars and needed to switch their assembly lines quickly from one run to another. Their emphasis wasn’t on making very specialized machines to do one task quickly; it was on producing general machines that could do a number of different things and could be switched quickly from doing one job to another. Toyota then does small runs on their assembly line rather than one massive long run. The big benefit to doing things in small batches was that quality problems are found much sooner. Hence they can stop the assembly line on the first car and fix a problem, rather than producing thousands of cars with the problem and then possibly not fixing the problem because it’s too expensive at that point. This also re-enforces the practice of fixing problems right away rather than letting them pile up and that it’s more cost effective to have the production line idle while the problem is fixed than having to fix it later.

Another example from the book “Lean Thinking” by James Womack and Daniel Jones recounts a story of one of the authors stuffing envelopes with their two young children. The children wanted to take the approach of folding all the letters, then putting all the folded letters in the envelopes then sealing all the envelopes then putting stamps on all the envelopes. This seems like an efficient and intuitive way to approach the problem. However the author insisted they do it the less intuitive way of one complete envelope at a time, i.e. folding one letter, stuffing it in an envelope, sealing it and putting a stamp on and then going on to the next one. Which way is faster? It turns out that researchers have timed groups of people performing this task and it turns out the one at a time approach is faster! It has a couple of other advantage as well. One is that if there is a problem, you find out right away; if the envelope is the wrong size, you find out on the first letter, not after you have folded all the letters, this could save you then having to refold all the letters to fit the envelopes. Similarly if you take a break you know how far along you are, if you’ve completed 40 out of 100 letters then you are 40% done and you can estimate your remaining time easily. You can’t do this when you do each step completely. This process is called “single piece flow” and is an example of doing a task as lots of small batches rather than one large batch. This is a general principle you want to apply in everything you do, resisting the old Ford type assembly line type mentality. You also receive the first finished product off the assembly line quicker, so you can validate it with your customer before too many more are produced to ensure you are producing the right thing.

Kanban vs Scrum

To some degree batch size is at the heart of the difference between the Kanban and Scrum methods of developing software. Scrum tends to batch things into sprints (usually two or three weeks long) which is much smaller than traditional waterfall development, but Kanban wants to take it further. In Kanban you only work on one or two small things at a time and finish one before starting the next things. You operate on lots of small batches, but you also have to limit how many batches (or tasks) you can have in progress at a time. This is called limiting work in progress (WIP). In Kanban you don’t worry about batching things up into sprints, just keeping the jobs small and limiting WIP. Kanban requires a lot of discipline and often if a development team hasn’t successfully progressed from waterfall to scrum, then adopting Kanban is a disaster.

The goal here is to produce small high quality increments that are immediately validated by customers. This way if you are going in the wrong direction, you won’t get too far before discovering it and not too much time will have been wasted (only a couple of small batches).

How do you get customer feedback, one small increment at a time? Obviously you can’t have dozens of customers download, install and play with a new version of your product every time you add a small increment and then interview them about it. But there are techniques you can use to do just this. The first is to produce your software via “continuous deployment”. This means that every build of your software is automatically deployed to customers whether that means pushing it to a web server or loading it on an automatic software updater service. You need to be able to control which customers get it and you need to be able to roll it back if something goes wrong. To be confident with this sort of procedure you need really good automated testing in place to ensure you haven’t accidentally broken something. Then to find out whether your increments (or batches) provide customer value, you need really good instrumentation in your product to report back on customer usage, whether they use the feature, how long it takes to perform tasks and any other useful statistics.

Software products are large and complicated systems. Managing that complexity is a major challenge. Breaking large tasks down into many small tasks is a powerful tool to reducing complexity. Combining many small but high quality parts results in emergent complexity, but again it is much easier to manage by adding one small thing at a time rather than suddenly throwing several large items into the mix at once.

Summary

Most of the ideas in this blog posting come from my reading the “Lean Startup” by Eric Ries over the holidays. Quite a good book with many useful ideas on product development and innovation. Reducing Batch size and limiting WIP appeal to me to help make Sage more agile and innovative as we move forward.

Written by smist08

January 8, 2012 at 12:35 am