Avoiding Agile Pitfalls
This article contains a number of pitfalls that I’ve run into as part of taking a large established development team from the Waterfall to the Agile Software Development Process. This involves taking a development organization with around 100 professionals in various roles including programming, quality assurance, business analysts, user centered design, product management, documentation, project managers, people managers, etc.
Through it all its best to always refer back to the Agile Manifesto as a guide to whether changes are helping or hurting:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Some of the items discussed here are controversial and I’ve certainly have colleagues who think these are good things. So keep in mind these are my personal experiences and judgments.
Too Much Process
Generally the solution to most problems with Waterfall involves adding more process, such as additional checkpoints and oversight. There is a huge tendency to carry this sort of thinking to Agile. It is totally alien that something can work without adding back all these layers and processes. Empowering teams to produce working software rather than documents and plans is always a challenge.
As people are new to Agile, they go through various training classes and read various books on the Agile process. To help with the learning they develop checklists, such as things to do at standup or things that need to be done to mark a story as done.
Creating a checklist of all the things that need to be done to complete a story seems like a good idea and usually has useful items like performing a code review or making sure unit tests are in place. However then blindly applying these checklists to all stories causes all sorts of unnecessary work. It also tends to set people up as gatekeeper which tends to be a bad thing. For instance a story that involves configuring some XML configuration files for a server component shouldn’t need any user centered design review, or Java code review. However when these are on the checklist, people blindly assume they need someone with these expertise to sign-off to complete the story. Or they need a special dispensation to skip the item. All of this leads to unnecessary discussion, meetings and wasted time. Worse it leads to feature trading negotiations. Say one of these reviewers doesn’t have anything relevant for this story, but want something low priority done somewhere else, and then they will “trade” their approval for this other work being done. This then subverts the product owner’s prioritization of the backlog.
In general these are symptoms of the product owners and scrum masters not having enough authority, trust or power. The scrum master must be diligently and aggressively removing check list items that aren’t appropriate for a situation. Management has to trust the scrum process and team empowerment to believe that the team will do the right thing and apply necessary done criteria, but leave off everything else.
Most agile books advocate creating a priority ordered backlog and then having teams swarm on the top items to get these most important items done first and done quickly. This is all great, but when you have a large development organization say with twelve scrum teams, then having this many people swarm on one problem leads to everyone tripping over everyone else.
I think you need to keep the whole swarming thing within one team. If you want multiple teams swarming on something, then there is something wrong with your backlog and it needs to somehow be refactored. Much more important than swarming is keeping things in the backlog independent of each other and really minimizing dependencies.
I think otherwise you get people thinking along the lines that if it takes a woman nine months to have a baby and Product Management has promised that baby in one month then you need to swarm on the problem and have nine mothers produce the baby in one month. Generally be careful that you aren’t really just slowing things down and distracting the people doing the real work by throwing resources at a problem.
Not Decomposing Problems Enough
There is a huge tendency for an organization that once did waterfall, when switching to agile, to keep doing waterfall, just fitting the tasks into sprints. A clear telltale for this is when they have a coding story that takes one sprint and then a QA story for the next sprint. This is really an attempt to separate programmers from QA and not have them working together on a story. The programmer is just throwing the code over the wall (or sprint end) to the QA just like the good old waterfall days.
The usual excuse for this is that the story is intrinsically to complicated or intertwined to break down and must be done over multiple sprints. Don’t believe this. Every case I’ve looked into, that did this, could be quite easily decomposed into multiple stories. It’s just a matter of adjusting your thinking to work that way.
I don’t like forcing rules on scrum teams, since this undermines their authority to do what is best for a story, but this is the one case where I find this necessary. If this isn’t done then the stories just get larger and larger until waterfall is back to being the norm.
I really emphasize releasable state, that these big stories are taking you out of releasable state and that this is a very bad things.
Another problem I’ve seen is that during sprints, teams are checking things madly into the main source tree so that the continuous build and integration system picks them up and their QA can immediately test what they have done. However this can be disruptive to other teams, if one team is breaking things mid-sprint.
If this starts happening, more use of branch/merge needs to be done. With Git this is very natural. With Subversion, this isn’t too hard either. In any case each team needs to cognizant of not disrupting other teams as they race to complete their own stories. Modern tools have all the features to facilitate this; you just need to educate people to use them. Basically if your changes are breaking other teams, then work in a branch and don’t merge until your testing is complete. With test driven development make sure the testing is in place and working before committing to the trunk.
These are some of the problems that I’ve run into with Agile. I find that as long as teams stick to the intent of the Agile Manifesto, then they can be extremely productive and produce very high quality work. The key is to empower the teams, stick to the basic Agile principles and avoid too much process or extra bureaucracy.