The Biggest Mistakes We Made on Our Agile Journey (and Why We are Glad We Made Them) — The Illusion of Velocity

Guest Blogger: Tim Coonfield
Sr. Director of Interconnected Retail Technology
Global Custom Commerce (a Home Depot Company)

I am working with Tom Cabanski to develop a talk for the one day Agile Shift conference scheduled for April 12, 2019 in Houston, TX titled “10 Biggest Mistakes We Made on Our Agile Journey (and Why We are Glad We Made Them)“. This is the eighth in a series of articles that Tom and I will use to explore some of those mistakes and what we learned from them. You can find other articles in this series here.

If you are interested in hearing this talk or some of the other awesome speakers and topics that will be covered at the event, you can learn more about the conference and purchase tickets here.

Everything I will share here happened at Global Custom Commerce (GCC), a Home Depot Company, as we developed and improved our 2nd generation e-commerce platform, called Autobahn, designed to make it easy for customers to shop for and buy more complex configurable products and services like custom window blinds, custom windows, flooring and decks. 
 
In 2011, when the Autobahn platform started development, GCC was already the #1 seller of custom window coverings online and owned several brands including blinds.com. We were a couple years away from acquisition by the Home Depot and had about 80 employees. The existing e-commerce platform had been in production for a number of years and was still actively being improved by a large team following a traditional project management philosophy using Gantt charts and reporting on percent complete.

The Autobahn project marked a number of firsts for GCC including our first use of cloud hosting at AWS and our first use of agile methodologies. This article highlights one of our bigger mistakes and how we were able to improve as a result.

As time winds down in the first half of a game, your team’s star point guard hurriedly dribbles the ball up court and heaves up a long three point attempt…just a hair after the buzzer sounds. He didn’t get the shot off on time. It misses anyway (as most 70-foot heaves do), and he and his team shrug it off as they walk to the locker room. What if I told you that he had no intention of getting that shot off on time, or that the half beat late shot was no accident at all?

This isn’t to demonize every NBA player, but all too often, it’s true. With the rise of sabermetrics (popularized in the 2003 novel, Moneyball) and other advanced measures of player success, seemingly innocuous missed shots can impact key metrics for a player that can ultimately cost them tens of thousands or even millions in eventual contract negotiations. With that in mind, it’s not unthinkable that a player would take the shot just after time expired, ensuring that it doesn’t actually count as a shot (and inevitable miss), thereby preserving their field goal percentage. Most of the time, it’s a harmless gaming of the statistics and little more than a forgotten anomaly on the stat sheet. On occasion, though, the desperation heave hits home and the team heads off the court three points short of where they could have been.

When I reflect on this, I can’t help but recall to an old adage of management: “You get what you measure.” As much as I can fault a player for not putting the team needs above their own self interest, human nature suggests that individuals will always optimize to the metrics that you put in front of them. It was with this truism that I look back on our spotted history of measuring success within our Autobahn development team.

The Autobahn project remains to this day the largest scale development effort we’ve ever completed at GCC. At the same time, it represented our first ever agile project and team. Under the scrum methodology, we quickly settled into a two-week sprint rhythm. Almost immediately, we started to struggle with the concept of velocity. Coming from a more traditional project management philosophy, leaders valued predictability in a plan above all else, and so in those early days, we looked to deliver that through our velocity. I don’t know where the number came from, but we settled on 30 points as our small team’s velocity. Without fail, we hit EXACTLY 30 points every single sprint. We would scale stories, counting hours and resizing stories – even assigning points to any non-feature tasks – to ensure that we were delivering precisely what the business had been conditioned to expect.

First big mistake. This way of looking at velocity inherently separated our sense of team value from our ability to deliver business value in any way. We were chasing the number. We wouldn’t stretch for additional stories beyond our 30 points. Our team worried more about how many points to assign to a technical chore or an all day training session than we would about positively impacting customer experiences. We were delivering 30 points without actually understanding whether we were getting closer to our goal.

The tipping point for this phase was when we literally assigned 3 points to a “story” in which we spent a half day cleaning and rearranging our workspace to prepare for an influential visitor to our offices. A few of us finally woke up and started to ask what the heck was going on.

The next several months was a litany of attempts through which our team iterated towards a healthier notion of team progress. We started more effectively planning chores and other tasks. We worked to understand the impact of these “zero business value” tasks to our ability to deliver. Our sprint reviews started to prominently feature burnup charts so we could understand team improvements in velocity and how our progress was moving us closer to our goal.

Enter the next phase – one that I know many teams have struggled with. Management throughout the company started asking how best we could measure the productivity of a development team. This was accentuated when we expended to 2 and 3 individual development teams. Anyone who’s been through an agile rollout at scale probably knows what’s coming, but inevitably my boss asked me, “Can you start reporting out each team’s independent velocity so we can compare effectively?”

I completely understood and appreciated the desire to better understand team performance and honestly had little doubt that the intent behind the request was completely constructive in its nature. That didn’t make the request any less problematic. Asking teams for a larger velocity – as EXTERNALLY measured – is a recipe for a myriad of unhealthy behaviors. Teams control their own destiny and can simply start changing 8 point stories into 13 point ones at a stroke of a pen. Instant velocity! Developers are motivated to push time consuming yet valuable technical chores to other teams so that they absorb the impact to their velocity. It’s not a matter of developers gaming the system – more often than not, it’s a subconscious reaction.

Many of us on the team immediately recognized how critical it was to maintain velocity as an internal metric for the team to understand their own growth and progress. I at the time fought hard to keep external readouts of velocity limited to our understanding of a project burnup chart and how we were progressing to business goals.

In the years since then, we have built some of the most talented and effective development teams that I have ever worked with, but we still struggle sometimes to quantify it exactly. The best thing I can say about our evolution as a business, though, is that we really don’t ask the question that much anymore. We’re more concerned with aligning groups of people – marketing, sales, development, etc. – around a single business goal. We all want to deliver business value and measure ourselves against the success of our business, not a story point or even cycle time metric.

We still look at all of those things, but almost exclusively internal to a team. I once went to a conference around agile metrics where the presenter posited the notion of plotting all of your proposed measures on a graph of both usefulness and “potential for evil.” It’s the idea that you need to consider the basketball player scenario – what potential negative behavior your metric can drive that is ultimately counterproductive to your team goals. When you start motivating teams with business goals and milestones rather than simply internal “productivity” metrics, you start to push the behaviors you actually want!