On Organizing Agile

Introduction

Generally Agile Programming is first taught as a very simple process, only involving some simple tools found in most kindergartens. The basic ideas of Agile Programming are quite simple and designed to make software development easier with less wasted work. However many people that adopt Agile find this transformed into quite a rigid bureaucratic system that isn’t very productive. What are a few things that can go wrong? How can you return to the simplicity of the original Agile goals? How can you track and manage large projects with many teams and dependencies in an Agile world?

Agile Manifesto

Agile was started by a group of programmers who came up with a simple set of guiding principles.

The purpose of this wasn’t to obliterate all the items on the right, it was to set some priorities. I think a lot of times when organizations adopt Agile, they follow this initially, but over time develop fixed ways of doing things and begin to have rationalizations that when they are doing the things on the right, they are really doing the things on the left. It takes a bit of organizational self-awareness to stay on guard against this.

Organizing Big Projects

A lot of simple courses on Agile focus on projects executed by one sprint team. Then to scale the process they wave their hands and just say you have multiple sprint teams executing off the backlog. Sounds good but tends to neglect a few factors. One is that different teams develop different expertise. Perhaps because they have a lot of experience in a given area or they have been tasked with specializing in a specific area like perhaps interaction design. Often in a large project it is very inefficient to have all the teams taking items from the same backlog. Often it’s better to have the teams specialized on separate parts of the project. People want to be masters of what they do, and trying to make everyone a jack of all trades just makes people unhappy. Further software is complex and subtle, having teams really know a specific area can be a huge help to move things ahead.

Once you start developing these specialization, you start introducing dependencies where one team will be waiting for another. Once you have dozens of teams, then you will need some sort of Project Management structure in place to track of this. A simple burn down of a backlog just doesn’t cut it.

Agile versus Waterfall

I find that the Agile versus Waterfall debate tends to indicate problems. People will say they won’t do something because it’s “Waterfall” and not “Agile”. But the truth is that the elements of Waterfall still need to be done. There has to be some program design, there has to be some architectural design, and there has to be some documentation. There is going to be integration and system testing. Just the goal is that these aren’t the main activities and they are generally done in much smaller increments than Waterfall.

When people refuse to do tasks because they are “Waterfall”, it is usually a warning sign that something important is going to be missed because someone dislikes a particular task. Generally your ears should perk up when you hear this argument being used.

Are You Really Agile?

Another problem area I see is when Agile teams become very non-agile. By this I mean they do sprint planning and then for the duration of the sprint absolutely refuse to do anything that wasn’t specified in sprint planning.

This can be good for getting the backlog reduced. But if priorities change in the backlog, it could mean they are now doing the wrong things. This usually means that discoveries and knowledge learned in a sprint isn’t used until the next sprint. Further if a customer issue comes up, it can look very bad to customers if their problem isn’t going to be even considered till next sprint planning.

Further if your product runs in the cloud, you need to be instantly responsive. In this case your teams do need to drop everything when your live system has a problem. You can’t let your cloud service be down until the next sprint.

Kanban

One commonly proposed solution to the above is to use Kanban. Here you don’t do sprint planning and always execute the top priority items in the backlog. There is further discipline to help make this work like limiting work in progress. To have Kanban work once you have a production system requires really strong product owner to avoid having all capacity going to the production system. Often this means holding a hard line on what is really important for customers, fixing minor problems or introducing more important new features.

Working with Off-Shore Teams

If you are working with off-shore teams then documentation and communications become much more important. You also need to be careful not to introduce bottlenecks that would stop them getting their work done (like waiting for a remote product owner to update/input backlog stories).

Most big multi-team projects usually have at least one off-shore team these days. In order to scale product development departments being able to work effectively across multiple locations in very different time zones becomes crucial.

Often if you are having problems with your Agile development process, then adding an off-shore component will magnify them and lead to much more serious problems.

Similarly teams are often not so co-located due to team members working at home, perhaps in a different city. There are many new technologies to help this work as I blogged on here.

Backlog Priorities

Setting backlog priorities is often quite controversial because of the mixed needs of different members of the product owner community. There has to be a balance of what is needed for customer, what is needed by dependent teams as well as architectural and operational issues. It takes a strong product owner with a clear understanding of these issues to set the priorities appropriately. Further the product owner needs the power to stick to his priorities and not to be continuously overridden by people escalating issues up the chain of command.

Summary

Keeping large projects Agile is a real challenge. Keeping Agile teams Agile over the years is a major challenge. People like routine and are always trying to get a nice regular fixed routine. This can be nice, but not entirely realistic as everyone tries to do things in Internet time while handling problems in cloud services in real time.