BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Why Agile Often Fails: No Agreed Metrics

Following
This article is more than 4 years old.

“You must embed real-time metrics from the very start of a program because they are nearly impossible to retrofit.”
—John Rossman, Think Like Amazon (2019)

Some years ago, I participated in a meeting of major organizations—some of them world-famous —about metrics. We all proudly shared what we were doing in the way of metrics. We all had metrics, albeit with different approaches. Some were elaborate and mathematical, while others were light and impressionistic. Some were focused on outputs, while others dwelt on surveys and outcomes. Some were mainly internal, while others were external. But we all agreed that some metrics were necessary. It was clear if you didn’t have metrics, you were in trouble.

Getty

Then someone asked the fateful question, that still haunts me: had any of us ever experienced a significant change in our organization’s behavior as a result of those metrics? We went around the room and the answers were unanimous: none of our organizations had ever changed their behavior significantly as a result of the wonderful sets of metrics that we had developed.

Changes had come about in our respective organizations, but not primarily as a result of the metrics. It turned out that changes depended on who was in the meeting room when the issue arose, what status they had, how much lobbying had been done, how much passion they brought to the subject and how effectively their views were communicated. Metrics were, of course, part of those conversations, but the metrics didn’t drive the decision-making. If the metrics didn’t confirm a particular viewpoint, reasons were generally found as to why the metrics hadn’t come out “right” and that other factors had to be taken into account.

In effect, the metrics in our organizations weren’t meaningful metrics at all. The metrics were being used to confirm the beliefs that we, the senior managers, had about the subject at hand. The numbers were an elaborate form of numeric public relations, not real metrics. (The subject at that time happened to be knowledge management, and my organization happened to be the World Bank, but it might have been any organization and any aspect of management, such as sales, marketing, social media—or Agile.)

Remember: this is also how metrics were used in pre-modern science in the days before Sir Frances Bacon in 1620! Experiments and metrics were used to confirm pre-existing beliefs, not to question “what everyone knows”. As a result, pre-modern science didn’t advance. In fact, it stagnated for thousands of years. Now bureaucratic management had fallen into the same trap. An organization without robust metrics is a very slow learner.

Metrics In Genuine Agile Management

Genuine Agile management is different. For instance, at Amazon, metrics are established in advance of every activity and specify what actions are expected to happen in ways that can be measured in real-time. If the metrics show that activity is not having the impact that was expected, action is required.

© 2019 Bloomberg Finance LP

Every activity is in effect a genuine scientific experiment focused on whether it is delivering the value to customers. No activity begins until those metrics are in place. Rigorous decision-making occurs if things turn out differently. Because the metrics are real-time metrics, the organization knows exactly what is happening on a daily basis. As John Rossman points out in Think Like Amazon (2019)

Today, you need real-time data, real-time monitoring, and real-time alarms when trouble is brewing—not lag-time metrics that hide the real issues for 24 hours or longer. Your business should operate like a nuclear reactor. If a problem arises, you need to be aware of it immediately."

There are many elements needed to make Agile metrics work properly, but one of the most important—and most neglected—is getting agreement in advance as to the metrics that will measure what every activity or initiative is expected to accomplish, so that action can be taken if it doesn’t accomplish that, or do more of it, if it does.

Four Levels Of Metrics

Metrics in organizations operate at four levels:

  1. A good idea”: an activity that is undertaken because enough influential people believe that it is likely to have some benefits. These are often, as Scrum.org points out, “really just conjectures about what customers might like (sometimes referred to as HiPPOs, or Highly Paid Person’s Opinions) ... The specification of the solution is vague and imprecise…” At worst, they are the hobby horse of some organizational faction promoting their own particular interest.
  2. An output”: something internal, measurable but not necessarily related to any external customer. This is better than a mere conjecture that it is a good idea, but still not getting the organization very far in terms of understanding the activity's value.
  3. An outcome”: something external such customer satisfaction in relation to value delivered. It is often subjective and vague and fuzzy. The Net Promoter Score, which fits into this category, has been shown to be positively correlated with actual impact and is certainly better than not having any measure, but its meaning can be ambiguous and difficult to read.
  4. The impact”: changes in customer behavior that the product or service is intended to elicit. This goes beyond merely whether the customer buys the product or service and may include measures of actions—or non-actions—that you would expect if the customer is truly delighted, such as timely availability of the item, speed of delivery, percentage of unexpected “hiccoughs” in delivery, absence of returns and complaints, re-purchases of the product and related products, responses to surveys recommendation of the products to other customers, and so on.

Amazon’s metrics operate at the fourth level and aim to measure impact. Organizations that don’t have these kinds of metrics and the associated behavioral norms in place to support them are flying blind. It shouldn’t come as a surprise that they frequently crash. Why don’t intelligent organizations measure impact? One reason is that thinking through impact in advance isn’t easy.

Thinking Through Impact In Advance Is Hard

Thinking through impact in advance is hard, but it's one of the rules of the road at Amazon. In fact, work on an activity or capability can’t start unless and until the team has figured out how it will measure customers' response. Amazon builds in customer metrics as a “forcing function” from the outset. Teams may spend weeks just thinking through the metrics.

The metrics serve as a kind of "forcing function." As Rossman notes:

A forcing function is a set of guidelines, restrictions, requirements, or commitments that “force,” or direct, a desirable outcome without having to manage all the details of making it happen. Forcing functions are a powerful technique used at Amazon to enforce a strategy or change or to get a difficult project launched.”

Why Doesn't Traditional Management Do This?

Why doesn’t this happen in traditional management? In big corporations, there are of course often detailed project plans showing cost, time, risk, benefits, and so on. When the activities are major, there will be big elaborate plans, but their success may only be apparent over long periods.

Politics often enters the process, as priorities are negotiated, and after some “horse-trading” the budgets for the year are fixed until the next major budget review. Projects often have long lead times and things can be going wrong for a long time without anyone outside the project proper being made aware of it. When things start to go astray in a huge project, the risk of failure is tied up with the fate and careers of powerful executives; as a result, bad news may be kept hidden and hard decisions don’t get taken.

Ten Steps That Amazon Takes To Measure Impact

Here are ten steps that Amazon takes to avoid these problems.

  1. An Obsession With Customer Value, Not Shareholder Value

Everything begins with an obsession with delivering value to customers. CEO Jeff Bezos announced this in his 1997 letter to shareholders and Amazon has relentlessly stuck to that obsession ever since. Shareholder value is the result, not the goal.

Executive compensation at Amazon is aligned with these principles. Executives have relatively modest salaries plus stock. Their compensation is not differentiated in terms of the performance of their own units. There is thus no use of or need for share buybacks to goose the short-term stock price: Amazon has too many good new ideas that it wants to use its resources for.

2. Shared responsibility for customer focus

As in all examples of genuine Agile management, life at Amazon is characterized by a pervasive customer-obsessed mindset. Customer obsession is not just for top management, sales, and marketing. Everyone is expected to be obsessed with knowing about and enhancing the impact of what they do for the customer. 

3. Customer-Focused Metrics

Customer obsession at Amazon is enabled and driven by customer-driven metrics. In most big organizations, customer data—where it exists at all—is spotty and soft, while financial data is hard and ubiquitous. Guess what drives decision-making? The hard financial numbers consistently trump any soft customer numbers.

Not so at Amazon, where real-time customer metrics are built into every aspect of the work. As a result, “customer value” wins battles that it wouldn’t win in firms run with a financially oriented mindset.

4.      The Metrics Are Built On A Narrative

At Amazon, metrics have to make sense. No significant new activity can be undertaken at Amazon unless and until there is an exhaustive management review of a six-page document explaining the activity as a narrative. As Rossman says:

Writing ideas and proposals in complete narratives results in better ideas, more clarity on the ideas, and better conversation on the ideas… The initiatives will be smaller and less risky."

The six-page narrative is supported by another document known as the PR/FAQ, which contains an imagined future press release describing the benefits customers are getting and answers to “frequently asked questions” about how the activity was developed. The metrics by which customer benefits of the activity will be measured in real-time are based on this narrative.

5. Activities Report To The Organization, Not The Unit

In most big organizations, in the absence of customer obsession, one can easily imagine that a planning process as at Amazon would quickly degenerate into a battle between units fighting for resources for “their” activities and “their” units. This doesn’t happen at Amazon. As Rossman points out, at Amazon:

Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say, ‘That’s not my job.’”

Unlike most big organizations, executives at Amazon aren’t accorded prestige or salary according to the size of their staffs or their budget.

6. The Organization Budgets Activities, Not Units

What often makes traditional budgeting so frustrating is that everyone knows that substantive issues about purpose and priorities are not being addressed in the budget discussion: the budget battles are a proxy for unresolved power struggles between competing units and the divergent viewpoints of senior managers.

Traditional budgeting often reflects, reinforces and aggravates the siloed organizational structure. Siloes often get in the way of delivering value to customers. Rather than having truly cross-functional teams, with end-to-end responsibilities to deliver value to customers, supposedly “Agile” teams may work within their own silo and produce outputs, for which they may get credit in the budgeting process, but which lead to no immediate customer benefit.

7.  Work Is Done In Small Teams Working In Short Cycles

As in all Agile management, work is done at Amazon to the extent possible in small, integrated autonomous multidisciplinary teams, known at Amazon as “two-pizza teams,” i.e. a team that you could in principle feed with two pizzas.

At Amazon, two-pizza teams work like semi-independent entrepreneurial hothouses. Insulated from the greater organization’s bureaucracy, the Two-Pizza Teams encourage ambitious leaders, provide opportunity, and instill a sense of ownership.”

8.      Budgeting Is A Subset Of Planning, Not Vice Versa

Budgeting at Amazon is a subset of an exhaustive planning process known as OP1 in which every activity is reviewed in terms of its contribution to value for customers. At Amazon, the reviews principally revolve around real-time customer-related metrics of individual activities, not about which unit gets how much money. That review is possible because each activity is evaluated at the very outset from the perspective of its eventual customer impact.

9.      Controlling Spending vs. Managing Impacts

Traditional budgeting is predicated on careful monitoring of spending, often in the absence of reliable information about what is happening on the more important issue of customer outcomes. Bad news is often kept from top management until it is too late to do anything about it.

At Amazon, the availability of real-time customer-related metrics at all levels ensures radical transparency. There is nowhere to hide. Everyone knows everything.

10.  Relation To Human Resources

The problems of traditional budgeting are often aggravated by the firm’s HR system, which may reflect and reinforce the siloed and hierarchical structure and budget, in which customer outcomes and impacts are not fully in the picture. Thus, individuals within each organizational silo tend to be evaluated and rewarded in terms of producing the budgeted level of outputs within the given budget envelope for their unit.

Amazon’s compensation structure is focused on incenting long-term enterprise value creation. The focus is on achievement (big and small) with responsibility for getting to yes which is everyone’s job.

And read also:

Seven Things A Highly Agile CEO Does: Jeff Bezos

Follow me on Twitter or LinkedInCheck out my website or some of my other work here