Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘agile manifesto

On Organizing Agile

with 6 comments

Introduction

Generally Agile Programming is first taught as a very simple process, only involving some simple tools found in most kindergartens. The basic ideas of Agile Programming are quite simple and designed to make software development easier with less wasted work. However many people that adopt Agile find this transformed into quite a rigid bureaucratic system that isn’t very productive. What are a few things that can go wrong? How can you return to the simplicity of the original Agile goals? How can you track and manage large projects with many teams and dependencies in an Agile world?

Agile Manifesto

Agile was started by a group of programmers who came up with a simple set of guiding principles.

Mini-manifesto1

The purpose of this wasn’t to obliterate all the items on the right, it was to set some priorities. I think a lot of times when organizations adopt Agile, they follow this initially, but over time develop fixed ways of doing things and begin to have rationalizations that when they are doing the things on the right, they are really doing the things on the left. It takes a bit of organizational self-awareness to stay on guard against this.

Organizing Big Projects

A lot of simple courses on Agile focus on projects executed by one sprint team. Then to scale the process they wave their hands and just say you have multiple sprint teams executing off the backlog. Sounds good but tends to neglect a few factors. One is that different teams develop different expertise. Perhaps because they have a lot of experience in a given area or they have been tasked with specializing in a specific area like perhaps interaction design. Often in a large project it is very inefficient to have all the teams taking items from the same backlog. Often it’s better to have the teams specialized on separate parts of the project. People want to be masters of what they do, and trying to make everyone a jack of all trades just makes people unhappy. Further software is complex and subtle, having teams really know a specific area can be a huge help to move things ahead.

Once you start developing these specialization, you start introducing dependencies where one team will be waiting for another. Once you have dozens of teams, then you will need some sort of Project Management structure in place to track of this. A simple burn down of a backlog just doesn’t cut it.

Agile versus Waterfall

I find that the Agile versus Waterfall debate tends to indicate problems. People will say they won’t do something because it’s “Waterfall” and not “Agile”. But the truth is that the elements of Waterfall still need to be done. There has to be some program design, there has to be some architectural design, and there has to be some documentation. There is going to be integration and system testing. Just the goal is that these aren’t the main activities and they are generally done in much smaller increments than Waterfall.

When people refuse to do tasks because they are “Waterfall”, it is usually a warning sign that something important is going to be missed because someone dislikes a particular task. Generally your ears should perk up when you hear this argument being used.

Are You Really Agile?

Another problem area I see is when Agile teams become very non-agile. By this I mean they do sprint planning and then for the duration of the sprint absolutely refuse to do anything that wasn’t specified in sprint planning.

This can be good for getting the backlog reduced. But if priorities change in the backlog, it could mean they are now doing the wrong things. This usually means that discoveries and knowledge learned in a sprint isn’t used until the next sprint. Further if a customer issue comes up, it can look very bad to customers if their problem isn’t going to be even considered till next sprint planning.

Further if your product runs in the cloud, you need to be instantly responsive. In this case your teams do need to drop everything when your live system has a problem. You can’t let your cloud service be down until the next sprint.

Kanban

One commonly proposed solution to the above is to use Kanban. Here you don’t do sprint planning and always execute the top priority items in the backlog. There is further discipline to help make this work like limiting work in progress. To have Kanban work once you have a production system requires really strong product owner to avoid having all capacity going to the production system. Often this means holding a hard line on what is really important for customers, fixing minor problems or introducing more important new features.

Working with Off-Shore Teams

If you are working with off-shore teams then documentation and communications become much more important. You also need to be careful not to introduce bottlenecks that would stop them getting their work done (like waiting for a remote product owner to update/input backlog stories).

droppedImage

Most big multi-team projects usually have at least one off-shore team these days. In order to scale product development departments being able to work effectively across multiple locations in very different time zones becomes crucial.

Often if you are having problems with your Agile development process, then adding an off-shore component will magnify them and lead to much more serious problems.

Similarly teams are often not so co-located due to team members working at home, perhaps in a different city. There are many new technologies to help this work as I blogged on here.

Backlog Priorities

Setting backlog priorities is often quite controversial because of the mixed needs of different members of the product owner community. There has to be a balance of what is needed for customer, what is needed by dependent teams as well as architectural and operational issues. It takes a strong product owner with a clear understanding of these issues to set the priorities appropriately. Further the product owner needs the power to stick to his priorities and not to be continuously overridden by people escalating issues up the chain of command.

Summary

Keeping large projects Agile is a real challenge. Keeping Agile teams Agile over the years is a major challenge. People like routine and are always trying to get a nice regular fixed routine. This can be nice, but not entirely realistic as everyone tries to do things in Internet time while handling problems in cloud services in real time.

Avoiding Agile Pitfalls

with 3 comments

Introduction

This article contains a number of pitfalls that I’ve run into as part of taking a large established development team from the Waterfall to the Agile Software Development Process. This involves taking a development organization with around 100 professionals in various roles including programming, quality assurance, business analysts, user centered design, product management, documentation, project managers, people managers, etc.

Through it all its best to always refer back to the Agile Manifesto as a guide to whether changes are helping or hurting:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Some of the items discussed here are controversial and I’ve certainly have colleagues who think these are good things. So keep in mind these are my personal experiences and judgments.

Too Much Process

Generally the solution to most problems with Waterfall involves adding more process, such as additional checkpoints and oversight. There is a huge tendency to carry this sort of thinking to Agile. It is totally alien that something can work without adding back all these layers and processes. Empowering teams to produce working software rather than documents and plans is always a challenge.

Checklists

As people are new to Agile, they go through various training classes and read various books on the Agile process. To help with the learning they develop checklists, such as things to do at standup or things that need to be done to mark a story as done.

Creating a checklist of all the things that need to be done to complete a story seems like a good idea and usually has useful items like performing a code review or making sure unit tests are in place. However then blindly applying these checklists to all stories causes all sorts of unnecessary work. It also tends to set people up as gatekeeper which tends to be a bad thing. For instance a story that involves configuring some XML configuration files for a server component shouldn’t need any user centered design review, or Java code review. However when these are on the checklist, people blindly assume they need someone with these expertise to sign-off to complete the story. Or they need a special dispensation to skip the item. All of this leads to unnecessary discussion, meetings and wasted time. Worse it leads to feature trading negotiations. Say one of these reviewers doesn’t have anything relevant for this story, but want something low priority done somewhere else, and then they will “trade” their approval for this other work being done. This then subverts the product owner’s prioritization of the backlog.

In general these are symptoms of the product owners and scrum masters not having enough authority, trust or power. The scrum master must be diligently and aggressively removing check list items that aren’t appropriate for a situation. Management has to trust the scrum process and team empowerment to believe that the team will do the right thing and apply necessary done criteria, but leave off everything else.

Over-Swarming

Most agile books advocate creating a priority ordered backlog and then having teams swarm on the top items to get these most important items done first and done quickly. This is all great, but when you have a large development organization say with twelve scrum teams, then having this many people swarm on one problem leads to everyone tripping over everyone else.

I think you need to keep the whole swarming thing within one team. If you want multiple teams swarming on something, then there is something wrong with your backlog and it needs to somehow be refactored. Much more important than swarming is keeping things in the backlog independent of each other and really minimizing dependencies.

I think otherwise you get people thinking along the lines that if it takes a woman nine months to have a baby and Product Management has promised that baby in one month then you need to swarm on the problem and have nine mothers produce the baby in one month.  Generally be careful that you aren’t really just slowing things down and distracting the people doing the real work by throwing resources at a problem.

Not Decomposing Problems Enough

There is a huge tendency for an organization that once did waterfall, when switching to agile, to keep doing waterfall, just fitting the tasks into sprints. A clear telltale for this is when they have a coding story that takes one sprint and then a QA story for the next sprint. This is really an attempt to separate programmers from QA and not have them working together on a story. The programmer is just throwing the code over the wall (or sprint end) to the QA just like the good old waterfall days.

The usual excuse for this is that the story is intrinsically to complicated or intertwined to break down and must be done over multiple sprints. Don’t believe this. Every case I’ve looked into, that did this, could be quite easily decomposed into multiple stories. It’s just a matter of adjusting your thinking to work that way.

I don’t like forcing rules on scrum teams, since this undermines their authority to do what is best for a story, but this is the one case where I find this necessary. If this isn’t done then the stories just get larger and larger until waterfall is back to being the norm.

I really emphasize releasable state, that these big stories are taking you out of releasable state and that this is a very bad things.

Source Control

Another problem I’ve seen is that during sprints, teams are checking things madly into the main source tree so that the continuous build and integration system picks them up and their QA can immediately test what they have done. However this can be disruptive to other teams, if one team is breaking things mid-sprint.

If this starts happening, more use of branch/merge needs to be done. With Git this is very natural. With Subversion, this isn’t too hard either. In any case each team needs to cognizant of not disrupting other teams as they race to complete their own stories. Modern tools have all the features to facilitate this; you just need to educate people to use them. Basically if your changes are breaking other teams, then work in a branch and don’t merge until your testing is complete. With test driven development make sure the testing is in place and working before committing to the trunk.

Summary

These are some of the problems that I’ve run into with Agile. I find that as long as teams stick to the intent of the Agile Manifesto, then they can be extremely productive and produce very high quality work. The key is to empower the teams, stick to the basic Agile principles and avoid too much process or extra bureaucracy.

Written by smist08

February 18, 2012 at 8:30 am