Stephen Smith's Blog

Musings on Machine Learning…

Posts Tagged ‘agile

Agile Vs Roadmap

with 9 comments

Introduction

We often receive RFPs (Request for Proposals) that demand a firm committed five year product roadmap. Similarly we are often criticized for not having such a “golden” roadmap when other competing products have. Having now worked in an Agile world for some time now, these requests seem stranger and stranger.

The quibble here isn’t with having a plan, it’s with the inflexibility these requests imply. That a company needs to set its course for five years and then any change in that plan is somehow a failure to deliver. That as knowledge and circumstances change that you need to stick to the plan and not adapt to the new situation.

Products are now introduced in “Internet” time. This means they are updated far more frequently (sometimes several times a day). All companies are looking to be “disruptive” and to “redefine” their market. Under these fast moving and fast changing conditions does it make sense to have a fixed long term roadmap?

On the other hand a product needs direction. A product needs long term thinking. You need to decide when to do something quick and dirty versus laying more groundwork and infrastructure to support future features. Stakeholders need to have an idea where a product is developing and what might be coming down the road.

There are quite a few types of roadmaps, there are technology roadmaps, feature roadmaps, release roadmaps, stop list roadmaps, marketing, strategy and many others. This articles generally applies to any of these.

Roadmap

Waste Not, Want Not

One of the key tenants of Agile Development is to reduce and if possible eliminate waste. Waste is any extra work that is being performed by team mates that doesn’t directly add value during the agile sprints. One main source of waste is doing too much and too detailed estimating. If you want a team to commit to estimates then they have to spend a lot of time working through those estimates so that they have the necessary precision. However when you do this, this work is often just waste, since then the work isn’t done due to changing priorities or another team does the work and insists on repeating the process or the project is postponed and when its resumed things have changed.

Roadmaps tend to generate a lot of wasted work. Once a company wants a roadmap that everyone is committed to, then far too much time will have been spent working on the estimates. The trick here is to be willing to accept inaccurate estimates. Many studies have shown that inaccurate WAG (Wild-Ass Guesses) type estimates aren’t really any less accurate than carefully constructed ones. All you need to know for building a roadmap is the order of magnitude of an item, not the details.

Detailed estimates are only done when the stories are going to be performed by the agile team. This work usually happens as a part of backlog grooming in the sprint before the work is actually going to be done. This then ensures that the stories are properly broken down and that the work can fit into one sprint.

Accept the Roadmap as a Guideline

The best way to think of a roadmap is as a guideline for current thinking. It is a mechanism to elicit feedback which can then be used to produce a better roadmap. Publishing a roadmap as a “fait accompli” doesn’t serve nearly as well as using a roadmap as a starting point for a conversation.

Often getting customer feedback on direction without providing any context or ideas is quite difficult because customers don’t spend their time thinking about how you need to develop your product. With a roadmap they can see how your product will fit in (or not) with their future business directions. Then they can provide useful feedback on what will be useful, what will be irrelevant and what will actually be harmful.

Keep in mind that conversations are two way things and the only way to be successful is to incorporate the feedback received and to show that the time spent by the customer talking to you is worthwhile. Corporations that can incorporate and synthesize the feedback from hundreds of customers in an effective manner tend to be the companies that really shine.

agile

Accept that Agile Works

A lot of times the push for a fixed roadmap is a result of the organization outside of R&D not being comfortable with the Agile idea of working on the most important story all the time. They liked the old days where a giant requirements document was produced and upper level management reviewed this and then felt comfortable that they could let R&D go off for a year or two to work on this without paying any more attention.

Generally it’s proven out that Agile is much more efficient and produces better products that meet customers’ needs much better than the old waterfall execute the requirements approach. But if upper management wants to know what R&D is doing they have to pay attention since things are fluid and always changing. This can be hard to accept, but now its being found that Agile can be applied to other parts of the organization and rather than older parts of the company dragging down the Agile parts, now all departments are going to Agile and its working very well for modern companies. In fact many people now believe that if a company doesn’t make this transition then it will become less and less competitive. The sad part is that Agile produces far better artifacts showing the progress of a project, you just need to learn how to use the tracking software to see them (another wastage is producing specialized reports just for upper management consumption).

Customer Connectedness

In the end, the goal is to be as customer connected as possible. Always working on the item in the product backlog with the highest value for the customer. This is now a proven principle for success. Dictating to customers what is good for them will just alienate your customers and send them elsewhere.

Creating a general roadmap that is used to get customer feedback and buy-in is just one tool of many to being a better customer connected company. And again the key secret ingredient is always adapting to change and not becoming fixed in your direction.

Of course when talking to customers and often other stakeholders, they will start out with how they need everything yesterday. But you have to steer the conversation to choosing priorities and not being brow-beaten into accepting that any estimates need to be shorter.

Summary

Roadmaps are great tools for having a conversation with stakeholders on the direction of a product. You just have to be careful not to fall into the trap that the roadmap is somehow a commitment that can never change. If done properly it can serve quite a few goals that are fully compatible with an Agile methodology.

If you are presenting a roadmap at a conference or WebEx, always prefix the presentation with that this is our current thinking and that we are always looking for feedback and ways to make the roadmap better.

 

Advertisements

Written by smist08

October 28, 2015 at 8:05 pm

Bangalore Travel Blog Part 2

with 3 comments

Introduction

This is part 2 of my travel blog on this trip to Bangalore, India. Part 1 is located here. In the first part I mostly covered the logistics of getting here. Now for this part I’ll continue with some of my impressions and learnings from being here. I was hosted on a very interesting day trip to Mysore and I’ve gotten quite a bit of exposure to living with the time zone difference and dealing with Indian traffic.

Time Zone

I’m from Vancouver, Canada which is in the Pacific (PST) time zone. Bangalore, India is on India Standard Time (IST). Basically IST is currently 13½ hours ahead of PST (in the summer during daylight savings time it’s only 12½ hours ahead). The reason for the ½ hour is that they didn’t want India to have two time zones so they took the average instead of dividing the country into two zones. This time zone difference is a big headache when working with Indian teams.

When in Vancouver my complaint is that during the winter I have to get to the office at 7:00am for various conference calls. Here I have the reverse problem. Now I have to get back to my hotel room for an 8:30pm conference call with Vancouver. Restaurants all open for dinner at 7pm here, so sometimes this can be tight.

I think it’s generally less disruptive to life, getting up earlier for these calls. I find it more disruptive having a big block of my evening consumed with them. Right now it doesn’t matter too much since I’m travelling and away from my family. But I’m pretty sure I would get some objections being on conference calls each evening between 8:30 and 10pm. So I really appreciate my Indian colleagues taking the evening side so I can take the morning side and I won’t complain so much in the future.

time_95

Agile Development and Indian Traffic

Traffic in India is very different from traffic in North America. Beyond the general busy-ness, a key difference is the variety of traffic on the road. It isn’t just cars and motorbikes (which can all maintain the same speed). In North America we get annoyed if a backhoe or cyclist is slowing down the flow of traffic. Here in India there are oxen towing carts, people pushing carts, horses, bicycles, tuk-tuks, overloaded slow moving trucks, vehicles that aren’t running properly, etc. Add to that cows and dogs deciding to randomly cross the street and lots of pedestrians crossing through the traffic. Generally it appears very chaotic where each vehicle moves ahead where it can, taking advantage of any open space there is. Everyone tries to flow around slow moving vehicles. Honking is very frequent to let people know you are there.

bangaloretraffic

Out of all this apparent chaos, traffic does move and you do get to where you want to go. To some degree there are un-written rules and the behavior of other drivers is quite expected to the locals. But the locals still shake their heads at drivers going the wrong way down divided roads or suddenly changing direction.

Agile development in India has a tendency to operate a bit like Indian traffic. You generally have more people on a project, so things are a lot busier. All the teams are jostling ahead, moving into open spots where they can (perhaps taking an easier backlog item to move ahead on).

This tends to mean that teams are very good at removing obstacles and making progress. Generally making any progress is considered good. Where this runs into trouble is when things need to be synchronized. Getting a good stable build for integration testing is very difficult because each team keeps nudging ahead and when you fix one problem, another is introduced.

Speed Humps

The way this is all controlled in traffic is via the strategic placement of speed humps. For instance if the road is divided, there are specific breaks in the divider where you are allowed to make a U-turn. But how do you do that into a continuous stream of oncoming traffic? Here they place a speed bump in the way of the oncoming traffic to slow it down and space it out, so that the people can merge in without having a head on collision. There aren’t many traffic lights here, but the strategic placement of obstacles seems to do the trick. The main downside of the speed humps is that they are rather severe and hard both on the people’s backs and on the car’s suspension.

speedhumps

Within the software development process you need to introduce the same sort of speed humps to control the flow of the project. This includes code review checkpoints, UI reviews, functional quality reviews and various other measures. Unless these measures have some teeth (like a fairly big triple speed hump), they will be ignored, but if they are enforced they do provide a good way to keep things synchronized. Indian project management is very focused on project metrics, so speed humps that affect the metrics get a lot of attention.

Honking

The most immediately noticeable difference between driving in India versus North America is all the honking. In North America honking is only used in extreme circumstances and people get irate (often leading to road rage) if they feel like they’ve been honked at unfairly. In India honking is a form of communication. There are subtle nuances as a quick tap of the horn is to let someone know you are there, a longer honk to say “watch out”. There are other forms of communication like flashing your lights (usually to mean move over so I can pass).

Similarly Indian development teams tend to be larger and there tends to be much more communications. Most of this is in the form of spirited discussions in the team areas. All this communication is good and I think we need more of this in North America. But beware like the communications between cars in traffic there are many local cues that non-Indians will not pick up on. One common reason for miscommunication is the Indian head nodding which is explained here. Remember that when you are having a conference call from North America you can’t pick up on all the body language that is going on and you could be missing and important aspect of the communications. For this reason having in person communications works better or consider using telepresence so you can pick up on these aspects.

Conclusion

This concludes part 2 of my coverage of my trip to Bangalore. Certainly it’s been an interesting trip so far and I’m picking up lots of insights in how things work.

Written by smist08

November 9, 2014 at 6:58 am

On Organizing Agile

with 6 comments

Introduction

Generally Agile Programming is first taught as a very simple process, only involving some simple tools found in most kindergartens. The basic ideas of Agile Programming are quite simple and designed to make software development easier with less wasted work. However many people that adopt Agile find this transformed into quite a rigid bureaucratic system that isn’t very productive. What are a few things that can go wrong? How can you return to the simplicity of the original Agile goals? How can you track and manage large projects with many teams and dependencies in an Agile world?

Agile Manifesto

Agile was started by a group of programmers who came up with a simple set of guiding principles.

Mini-manifesto1

The purpose of this wasn’t to obliterate all the items on the right, it was to set some priorities. I think a lot of times when organizations adopt Agile, they follow this initially, but over time develop fixed ways of doing things and begin to have rationalizations that when they are doing the things on the right, they are really doing the things on the left. It takes a bit of organizational self-awareness to stay on guard against this.

Organizing Big Projects

A lot of simple courses on Agile focus on projects executed by one sprint team. Then to scale the process they wave their hands and just say you have multiple sprint teams executing off the backlog. Sounds good but tends to neglect a few factors. One is that different teams develop different expertise. Perhaps because they have a lot of experience in a given area or they have been tasked with specializing in a specific area like perhaps interaction design. Often in a large project it is very inefficient to have all the teams taking items from the same backlog. Often it’s better to have the teams specialized on separate parts of the project. People want to be masters of what they do, and trying to make everyone a jack of all trades just makes people unhappy. Further software is complex and subtle, having teams really know a specific area can be a huge help to move things ahead.

Once you start developing these specialization, you start introducing dependencies where one team will be waiting for another. Once you have dozens of teams, then you will need some sort of Project Management structure in place to track of this. A simple burn down of a backlog just doesn’t cut it.

Agile versus Waterfall

I find that the Agile versus Waterfall debate tends to indicate problems. People will say they won’t do something because it’s “Waterfall” and not “Agile”. But the truth is that the elements of Waterfall still need to be done. There has to be some program design, there has to be some architectural design, and there has to be some documentation. There is going to be integration and system testing. Just the goal is that these aren’t the main activities and they are generally done in much smaller increments than Waterfall.

When people refuse to do tasks because they are “Waterfall”, it is usually a warning sign that something important is going to be missed because someone dislikes a particular task. Generally your ears should perk up when you hear this argument being used.

Are You Really Agile?

Another problem area I see is when Agile teams become very non-agile. By this I mean they do sprint planning and then for the duration of the sprint absolutely refuse to do anything that wasn’t specified in sprint planning.

This can be good for getting the backlog reduced. But if priorities change in the backlog, it could mean they are now doing the wrong things. This usually means that discoveries and knowledge learned in a sprint isn’t used until the next sprint. Further if a customer issue comes up, it can look very bad to customers if their problem isn’t going to be even considered till next sprint planning.

Further if your product runs in the cloud, you need to be instantly responsive. In this case your teams do need to drop everything when your live system has a problem. You can’t let your cloud service be down until the next sprint.

Kanban

One commonly proposed solution to the above is to use Kanban. Here you don’t do sprint planning and always execute the top priority items in the backlog. There is further discipline to help make this work like limiting work in progress. To have Kanban work once you have a production system requires really strong product owner to avoid having all capacity going to the production system. Often this means holding a hard line on what is really important for customers, fixing minor problems or introducing more important new features.

Working with Off-Shore Teams

If you are working with off-shore teams then documentation and communications become much more important. You also need to be careful not to introduce bottlenecks that would stop them getting their work done (like waiting for a remote product owner to update/input backlog stories).

droppedImage

Most big multi-team projects usually have at least one off-shore team these days. In order to scale product development departments being able to work effectively across multiple locations in very different time zones becomes crucial.

Often if you are having problems with your Agile development process, then adding an off-shore component will magnify them and lead to much more serious problems.

Similarly teams are often not so co-located due to team members working at home, perhaps in a different city. There are many new technologies to help this work as I blogged on here.

Backlog Priorities

Setting backlog priorities is often quite controversial because of the mixed needs of different members of the product owner community. There has to be a balance of what is needed for customer, what is needed by dependent teams as well as architectural and operational issues. It takes a strong product owner with a clear understanding of these issues to set the priorities appropriately. Further the product owner needs the power to stick to his priorities and not to be continuously overridden by people escalating issues up the chain of command.

Summary

Keeping large projects Agile is a real challenge. Keeping Agile teams Agile over the years is a major challenge. People like routine and are always trying to get a nice regular fixed routine. This can be nice, but not entirely realistic as everyone tries to do things in Internet time while handling problems in cloud services in real time.

The Road to DevOps Part 1

with 5 comments

Introduction

Historically Development and Operations have been two separate departments that rarely talk to each other. Development produces a product using some sort of development methodology and when they’ve finished it, they release it and hand it off to Operations. Operations then installs the product, upgrades the users from the old version and then maintains the system, fixing hardware problems, logging software defect complaints and keeping backups. This worked fine when Development produced a release every 18 months or so, but doesn’t really work in the modern Web world where software can often be released hourly.

In the new world Development and Operations need to be combined into one team and one department. The goal being to remove any organizational or bureaucratic barriers between the two. It also recognizes that the goal of the company isn’t just to produce the software but to run it and to keep it available, up to date and healthy for all its customers. Operations has to provide feedback immediately to Development and Development has to quickly address any issues and provide updates quickly back to Operations.

In this posting I’m covering the aspect of DevOps concerned with frequently rolling out new versions to the cloud. In a future posting I’ll cover the aspects of DevOps concerned with the normal running of a cloud based service, like provisioning new users, monitoring the system, scaling the system and such.

Agile to DevOps

We have transitioned our software development methodology from Waterfall to Agile. A key idea of Agile is that you break work into small stories that you complete in a single short sprint (in our case two weeks). A key goal is that during the sprint each story is done completely including testing, documentation and anything else required. This then leads to the product being in a releasable state at the end of each sprint. In an ideal world you could then release (or deploy) the product at the end of each sprint.

This is certainly an improvement over Waterfall where the product is only releasable every year or 18 months or so. But generally with Agile this is the ideal, but not really what is quite achieved. Generally a release consists of the outcome of a number of sprints (usually around 10) followed by a short regression, held concurrently with a beta test followed by release.

So the question is how do we remove this extra overhead and release continuously (or much more frequently)? This is where DevOps comes in. It is a methodology that has been developed to extend Agile Development and principles straight through to actually encompassing the deployment and maintenance of the product. DevOps does require Development change the way it does things as it requires Operations to change. DevOps requires a much more team bases approach to doing things and requires organizational boundaries be removed. DevOps requires a razor focus on automating all manual processes in the build to deploy to monitor to get feedback cycle.

Development

Most development processes have an automated build system, usually built on something like Jenkins. The idea is that when you check in source code, the build system sees you checked it in, then it has rules that tell it what module it is part of, and rebuilds those modules, then it sees what modules depend on those modules and rebuilds those and so on. As part of the build process, unit tests are run and various automated tests are set off. The idea is that if anything goes wrong, it is very clear which check-in caused it and things can be quickly fixed and/or rolled back.

This is a good starting point but for effective DevOps it needs to be refined. Most modern source control systems support branching (most famously git). For DevOps it becomes even more crucial that the master branch of the product is always in releasable state and can be deployed at any time. The way this is achieved is by developing each feature as a separate branch. Then when a feature is completely ready for release it can be pulled into the master branch, which means it can be deployed at any time. Below is a diagram of how this process typically works:

Automated Testing

Obviously in this environment, it isn’t going to work if for every frequent release, you need to run a complete thorough manual test of the entire product. In an ideal world you have very complete test coverage using various levels of automated testing. I covered some aspects of this here. You still want to do some manual testing to ensure that things still feel right, but you don’t want to have to be repeating a lot of functional testing.

Operations

Operations can then take any release and in consultation with the various stakeholders release it to production. Operations is then in charge of this part of things and ensures the new versions in installed, data is converted and everything is running smoothly.

Some organizations release everything that is built which means their web site can be updated hourly. Github is a good example of this. But generally for ERP or CRM type software we want to be a bit more controlled. Generally there is a schedule of releases (perhaps one release every two weeks) and then a schedule of when things need to be done to be included in that release, which then controls which branches get pulled into the master branch. This is to ensure that there aren’t any disruptions to business customers. Again you can see that this process is blending elements of QA along with Operations which is why the DevOps team approach works so well.

A key idea that has emerged is the concept: “Infrastructure as Code”. In this world all Operations tasks are performed by code. This way things become much more repeatable and much more automated. It’s really this whole idea that you build your infrastructure by writing programs that then do the real work, that has largely led to movement of Developers into Operations.

DevOps

This is where Development and Operations must be working as a team. Operations has to let Developers know what tools (scripts) they require to deploy the new version. They need automated procedures to roll out the new version, convert data and such. They have to be working together very closely to develop all these scripts in the various tools like Jenkins, PowerShell, Maven, Ant, Chef, Puppet or Nexus.

Performing all this work takes a lot of effort. It has to be people’s full time job, or it just won’t get done properly. If people aren’t fully applied to this, manual processes will start to creep in, productivity and quality will suffer.

Beyond successfully deploying the software, this team has to handle things when they go wrong. They need to be able to rollback a version. Reverse the database schema changes and return to a known stable good state.

Summary

DevOps is a whole new profession. Combining many of the skills of Development with those of Operations. People with these skill sets will be in high demand as this is becoming an area that is making or breaking companies on the Web and in the Cloud. No one likes to have outages; no one likes to roll out bad upgrades. In today’s fast paced world, these can put huge pressures on a company. DevOps as a profession and set of operating procedures is a good way to alleviate this pressure while keep up to the fast pace.

Avoiding Agile Pitfalls

with 3 comments

Introduction

This article contains a number of pitfalls that I’ve run into as part of taking a large established development team from the Waterfall to the Agile Software Development Process. This involves taking a development organization with around 100 professionals in various roles including programming, quality assurance, business analysts, user centered design, product management, documentation, project managers, people managers, etc.

Through it all its best to always refer back to the Agile Manifesto as a guide to whether changes are helping or hurting:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Some of the items discussed here are controversial and I’ve certainly have colleagues who think these are good things. So keep in mind these are my personal experiences and judgments.

Too Much Process

Generally the solution to most problems with Waterfall involves adding more process, such as additional checkpoints and oversight. There is a huge tendency to carry this sort of thinking to Agile. It is totally alien that something can work without adding back all these layers and processes. Empowering teams to produce working software rather than documents and plans is always a challenge.

Checklists

As people are new to Agile, they go through various training classes and read various books on the Agile process. To help with the learning they develop checklists, such as things to do at standup or things that need to be done to mark a story as done.

Creating a checklist of all the things that need to be done to complete a story seems like a good idea and usually has useful items like performing a code review or making sure unit tests are in place. However then blindly applying these checklists to all stories causes all sorts of unnecessary work. It also tends to set people up as gatekeeper which tends to be a bad thing. For instance a story that involves configuring some XML configuration files for a server component shouldn’t need any user centered design review, or Java code review. However when these are on the checklist, people blindly assume they need someone with these expertise to sign-off to complete the story. Or they need a special dispensation to skip the item. All of this leads to unnecessary discussion, meetings and wasted time. Worse it leads to feature trading negotiations. Say one of these reviewers doesn’t have anything relevant for this story, but want something low priority done somewhere else, and then they will “trade” their approval for this other work being done. This then subverts the product owner’s prioritization of the backlog.

In general these are symptoms of the product owners and scrum masters not having enough authority, trust or power. The scrum master must be diligently and aggressively removing check list items that aren’t appropriate for a situation. Management has to trust the scrum process and team empowerment to believe that the team will do the right thing and apply necessary done criteria, but leave off everything else.

Over-Swarming

Most agile books advocate creating a priority ordered backlog and then having teams swarm on the top items to get these most important items done first and done quickly. This is all great, but when you have a large development organization say with twelve scrum teams, then having this many people swarm on one problem leads to everyone tripping over everyone else.

I think you need to keep the whole swarming thing within one team. If you want multiple teams swarming on something, then there is something wrong with your backlog and it needs to somehow be refactored. Much more important than swarming is keeping things in the backlog independent of each other and really minimizing dependencies.

I think otherwise you get people thinking along the lines that if it takes a woman nine months to have a baby and Product Management has promised that baby in one month then you need to swarm on the problem and have nine mothers produce the baby in one month.  Generally be careful that you aren’t really just slowing things down and distracting the people doing the real work by throwing resources at a problem.

Not Decomposing Problems Enough

There is a huge tendency for an organization that once did waterfall, when switching to agile, to keep doing waterfall, just fitting the tasks into sprints. A clear telltale for this is when they have a coding story that takes one sprint and then a QA story for the next sprint. This is really an attempt to separate programmers from QA and not have them working together on a story. The programmer is just throwing the code over the wall (or sprint end) to the QA just like the good old waterfall days.

The usual excuse for this is that the story is intrinsically to complicated or intertwined to break down and must be done over multiple sprints. Don’t believe this. Every case I’ve looked into, that did this, could be quite easily decomposed into multiple stories. It’s just a matter of adjusting your thinking to work that way.

I don’t like forcing rules on scrum teams, since this undermines their authority to do what is best for a story, but this is the one case where I find this necessary. If this isn’t done then the stories just get larger and larger until waterfall is back to being the norm.

I really emphasize releasable state, that these big stories are taking you out of releasable state and that this is a very bad things.

Source Control

Another problem I’ve seen is that during sprints, teams are checking things madly into the main source tree so that the continuous build and integration system picks them up and their QA can immediately test what they have done. However this can be disruptive to other teams, if one team is breaking things mid-sprint.

If this starts happening, more use of branch/merge needs to be done. With Git this is very natural. With Subversion, this isn’t too hard either. In any case each team needs to cognizant of not disrupting other teams as they race to complete their own stories. Modern tools have all the features to facilitate this; you just need to educate people to use them. Basically if your changes are breaking other teams, then work in a branch and don’t merge until your testing is complete. With test driven development make sure the testing is in place and working before committing to the trunk.

Summary

These are some of the problems that I’ve run into with Agile. I find that as long as teams stick to the intent of the Agile Manifesto, then they can be extremely productive and produce very high quality work. The key is to empower the teams, stick to the basic Agile principles and avoid too much process or extra bureaucracy.

Written by smist08

February 18, 2012 at 8:30 am

Doing Work in Small Batches and Limiting Work in Progress

with 5 comments

Recently Software Development has been borrowing a lot of ideas from Japanese Lean Manufacturing. In a way this mirrors the progression from the Detroit way of building cars pioneered by Henry Ford to the modern way pioneered by Toyota.

The original Ford production line approach was that a car was assembled on an assembly line by going through a series of specialized stations that were optimized to do one things really well and efficiently, so the car would go from perhaps the attach doors stations to the add Window glass station. Each station was optimized to do its job really well and the priority was to keep the assembly line moving no matter what. If the assembly line ever stopped then the hundreds of people working on it would be idled and this was considered a disaster. If problems occurred then they were noted and fixed post-assembly line. This process was then improved by optimizing each station to be quicker and more automated. Generally this produced a very good system of producing a very high volume of identical cars.

From the outside a Toyota assembly line looks nearly identical. But it operates very differently. Toyota didn’t have the same large market as Detroit and didn’t need the volume. They needed to produce more specialized cars and needed to switch their assembly lines quickly from one run to another. Their emphasis wasn’t on making very specialized machines to do one task quickly; it was on producing general machines that could do a number of different things and could be switched quickly from doing one job to another. Toyota then does small runs on their assembly line rather than one massive long run. The big benefit to doing things in small batches was that quality problems are found much sooner. Hence they can stop the assembly line on the first car and fix a problem, rather than producing thousands of cars with the problem and then possibly not fixing the problem because it’s too expensive at that point. This also re-enforces the practice of fixing problems right away rather than letting them pile up and that it’s more cost effective to have the production line idle while the problem is fixed than having to fix it later.

Another example from the book “Lean Thinking” by James Womack and Daniel Jones recounts a story of one of the authors stuffing envelopes with their two young children. The children wanted to take the approach of folding all the letters, then putting all the folded letters in the envelopes then sealing all the envelopes then putting stamps on all the envelopes. This seems like an efficient and intuitive way to approach the problem. However the author insisted they do it the less intuitive way of one complete envelope at a time, i.e. folding one letter, stuffing it in an envelope, sealing it and putting a stamp on and then going on to the next one. Which way is faster? It turns out that researchers have timed groups of people performing this task and it turns out the one at a time approach is faster! It has a couple of other advantage as well. One is that if there is a problem, you find out right away; if the envelope is the wrong size, you find out on the first letter, not after you have folded all the letters, this could save you then having to refold all the letters to fit the envelopes. Similarly if you take a break you know how far along you are, if you’ve completed 40 out of 100 letters then you are 40% done and you can estimate your remaining time easily. You can’t do this when you do each step completely. This process is called “single piece flow” and is an example of doing a task as lots of small batches rather than one large batch. This is a general principle you want to apply in everything you do, resisting the old Ford type assembly line type mentality. You also receive the first finished product off the assembly line quicker, so you can validate it with your customer before too many more are produced to ensure you are producing the right thing.

Kanban vs Scrum

To some degree batch size is at the heart of the difference between the Kanban and Scrum methods of developing software. Scrum tends to batch things into sprints (usually two or three weeks long) which is much smaller than traditional waterfall development, but Kanban wants to take it further. In Kanban you only work on one or two small things at a time and finish one before starting the next things. You operate on lots of small batches, but you also have to limit how many batches (or tasks) you can have in progress at a time. This is called limiting work in progress (WIP). In Kanban you don’t worry about batching things up into sprints, just keeping the jobs small and limiting WIP. Kanban requires a lot of discipline and often if a development team hasn’t successfully progressed from waterfall to scrum, then adopting Kanban is a disaster.

The goal here is to produce small high quality increments that are immediately validated by customers. This way if you are going in the wrong direction, you won’t get too far before discovering it and not too much time will have been wasted (only a couple of small batches).

How do you get customer feedback, one small increment at a time? Obviously you can’t have dozens of customers download, install and play with a new version of your product every time you add a small increment and then interview them about it. But there are techniques you can use to do just this. The first is to produce your software via “continuous deployment”. This means that every build of your software is automatically deployed to customers whether that means pushing it to a web server or loading it on an automatic software updater service. You need to be able to control which customers get it and you need to be able to roll it back if something goes wrong. To be confident with this sort of procedure you need really good automated testing in place to ensure you haven’t accidentally broken something. Then to find out whether your increments (or batches) provide customer value, you need really good instrumentation in your product to report back on customer usage, whether they use the feature, how long it takes to perform tasks and any other useful statistics.

Software products are large and complicated systems. Managing that complexity is a major challenge. Breaking large tasks down into many small tasks is a powerful tool to reducing complexity. Combining many small but high quality parts results in emergent complexity, but again it is much easier to manage by adding one small thing at a time rather than suddenly throwing several large items into the mix at once.

Summary

Most of the ideas in this blog posting come from my reading the “Lean Startup” by Eric Ries over the holidays. Quite a good book with many useful ideas on product development and innovation. Reducing Batch size and limiting WIP appeal to me to help make Sage more agile and innovative as we move forward.

Written by smist08

January 8, 2012 at 12:35 am

On Customer Adoption

with 3 comments

The Problem

Traditionally with ERP systems, a new version is released every 18 months or so. Typically customers will only upgrade every few versions for a number of reasons discussed below. This leads to a major problem, namely if a customer requests a new feature and we jump on it, then it still could be four years before they see this feature. How this usually works is shown in the following timeline.

In an agile on-line world, this seems like rather a slow process to provide value to customers. Most Sage products have an on-line suggestion web site, such as https://www11.v1ideas.com/SageERPAccpac/Accpac for Sage 300 ERP. However customers tend to give up on making suggestions to these once they enter a number of suggestions and don’t see any movement in the next few months. As more and more people are part of the MTV Generation (or later) then instant gratification becomes the expectation.

Traditionally this is all part of a fairly large and involved Product Development Life Cycle. Product Managers spend their time talking to customers and partners as well as evaluating various industry trends to come up with a list of features for the next version. Usually a number of themes are developed and implemented as major new features. R&D then takes as many such themes and features that they think this can develop in 18 months. If all goes well, these are implemented, extensive testing is performed and an updated version of the product ships after these 18 months. Then if Product Management has done their job properly and R&D has faithfully implemented what was laid out, then customers should be fairly happy with the new version.

Of course things aren’t quite that rigid anymore. Most development groups (including Sage ERP) have adopted the Agile Software Development Lifecycle where the development team is implementing features in small batches taken from a product backlog. The product backlog is in priority order so the higher priority features are implemented first. A side effect of this is that for lower priority items in the backlog, Product Management can swap these out with other higher priority features as business needs or understanding changes. This then leads to greater flexibility and does allow some newer customer suggestions to make it into a product sooner. I blogged on some aspects of this here and here.

Often we are tasked with the goal of shortening our release cycles; say from 18 months to 12 months or even 9 months. However we get a lot of resistance from customers and partners on this. The main reason is that traditionally upgrades are fairly costly and time consuming activities that can often be quite disruptive to a business. As a result customers typically don’t want to upgrade more than every three years, so there is no increase in customer value by providing more frequent releases. Even then the main reason customer’s site that they upgrade is to stay in support, since we only support the two most recent versions or to stay compatible with upgrades from other vendors, notably Microsoft. If you need to know you work well on Windows 7, then you need a more recent version of your products. A large part of this problem has been caused by us software vendors, when presented with a choice to add a new feature to a release or to make upgrading easier, we consistently choose to add the new feature and leave any difficulty with upgrading to the Business Partners to deal with; who, of course, pass this cost on to our customers. Right now 18 months works well because by supporting two versions, customers only need to upgrade every 3 years.

So out of this we have two significant problems to address:

  1. Making upgrades easier so it isn’t a huge inconvenience and cost to upgrade.
  2. Make the features and functionality we add to our products more compelling so customers cite these as the reason for upgrading and not just keeping up with Microsoft compatibility problems.

Frictionless Upgrades

To address the first point, we’ve been working hard to make the upgrade process easier. This is Sage’s Frictionless Upgrade initiative. We’ve just begun on this journey, but in a number of products we are already seeing results. Such as in Sage 300 ERP we’ve combined all the separate module installations into one master product installations, allow you to activate multiple applications to the new version in one go and to minimize database changes so Crystal Reports and Macros don’t need modifications.

Going Online with services like www.accpaconline.com can help, since now you don’t need to install the software yourself, but you still have the work of making sure custom reports still work and training staff on any changes to their workflow. Generally the process of upgrading can be done well or badly whether you are online or on-premise. Certainly we’ve seen cases where overnight changes to web sites like Facebook have been very disruptive.

Many products such as Windows or Apple iTunes have an auto-update feature, so even though these are on-premise installed on the local computer type applications, they can still be upgraded frequently with little fuss. Expect to see this sort of functionality come to more and more business applications allowing the software to be upgraded frictionlessly without requiring a major database conversion or requiring changes to customizations.

Making Features Compelling

The intent with all the customer visits, customer interviews and other research that Product Management performs is that the features we add to the product are compelling either to existing users or to attract new users. A lot of work goes into researching and collecting all the data that goes into this decision making. Generally this yields better results than being driven by technology trends or by what competitors are doing; however, we tend to not being seeing the results we would like. What is missing?

With the scientific method, like what you see in Physics, Chemistry or Biology, people come up with theories, but these aren’t really believed until they are backed by experimental evidence. In software development we tend to do a lot of work doing research and creating theories and spend a lot of time discussing these theories with customers, partners and analysts. However we don’t take the next step and scientifically test these theories in real customer situations.

We tend to rely on doing studies on customers using prototypes or mockups. But when customers aren’t in their real work environment, doing real work, the results tend to be suspect. Is the customer just telling you they like your new feature to make you feel good, even though they will probably never use it? Are the questions being asked at interviews slanted to get the answers the researcher wants?

I recently read “The Lean Startup” by Eric Ries. This book, among other things, heavily recommends real scientific experiments with real customers in their real environment. No more separate studies with proof of concepts or mock ups. In a way we are doing this now. The only problem is that we don’t know the result until after four years and then it’s very hard to remove something that really didn’t work.

So how do we perform these experiments on real users in their real workplaces doing real work? Sure it’s easy for Facebook to try a new feature in say one town to see what the reaction is, but here we are talking about real business software that people rely on to run their businesses.

The real trick to this is to keep features small and only try small changes at a time. This is often referred to as a Minimum Viable Product or MVP. Only the absolute minimum functionality to do the job is produced. Any additional functionality is only developed (and tested) due to customer demand. This tends to greatly reduce complexity and unnecessary bells and whistles that we see in so much software. This then allows features to be developed and tested quicker. In manufacturing this is referred to as keeping batch size small.

This requires a great deal of good instrumentation in the product, so we know if a feature is being used and if it is helping (are users really more productive entering invoices?). Creating a full release with all the changes from the work produced by a large team over a year is a major undertaking. But adding small changes that have been fully QA’ed is fairly easy. In fact we already do this with hotfixes. In a way each hotfix is an experiment as to whether we have fixed a defect (hopefully mostly we have). So why are small new features any different than hotfixes?

You might argue that SaaS vendors can do this easier, since they can introduced the change for just a few customers by routing them to a different application server. However with auto-update we could do the same for on-premise customers. Give them the new feature and then based on the results, if they are good, roll it out to all customers, but if they are bad, then roll it back for anyone that received it.

These techniques have produced great results in a number of well documented cases, but will they be accepted for business software? I think right now there will be a fair bit of resistance to these sort of practices, but I think it is really a PR exercise to prove to customers that by participating in these sort of experiments, that they will get the features they are asking for sooner and that the new features they receive will be far more valuable. One of the main advantages of these techniques is reducing waste, by detecting things that won’t work quickly and discarding them quickly. This way some really dumb idea can’t make its way all the way through development using all those resources, to just annoy users in the end. Chances are this will start with new connected services or new on-line features, but I think eventually this could become more mainstream for even large ERP and CRM systems.

Summary

New techniques are being developed to develop customer value quickly and to get it into customer’s hands much faster than we have traditionally. However some of these techniques like experimenting on customers running live systems are going to take a bit of good PR to prove their value to get the necessary participation. If this all sounds crazy, give “The Lean Startup” a read, it paints a fairly compelling picture.

Written by smist08

January 1, 2012 at 9:49 pm