In the movie Moneyball, about how the Oakland A’s became a winning baseball team by applying analytics and new types of metrics to the game, there is a delightful scene in which grizzled old scouts are sitting around a table discussing players.
The audience can’t help cringing, not simply because the scene exposes the way decisions are made devoid of data, but because we’ve all been in situations where “certainty” was based on sentiment rather than science.
The scene perfectly depicts the shortcomings of human judgment. What passes for reasoned debate is really based on nothing concrete. Decisions about millions of dollars’ worth of player contracts are made on gut instinct, absent of objective measures. Yes, it is just a film, but real life isn’t much different. Similar empty reasoning is employed from Manhattan boardrooms to the Oval Office to coffee shops and kitchen tables everywhere else.
Moneyball, based on the book by Michael Lewis, tells the true story of Billy Beane, the Oakland A’s general manager who threw out the century-old rulebook on how to value players in favor of a math- infused method that looks at the game from a new set of metrics.... Ultimately he led the long-suffering team to a first-place finish in the American League West in the 2002 season, including a 20-game winning streak. From then on, statisticians supplanted the scouts as the sport’s savants. And lots of other teams scrambled to adopt sabermetrics themselves.
In the same spirit, the biggest impact of big data will be that data-driven decisions are poised to augment or overrule human judgment.
Data-driven decisions are poised to augment or overrule human judgment.
The subject-area expert, the substantive specialist, will lose some of his or her luster compared with the statistician and data analyst, who are unfettered by the old ways of doing things and let the data speak. This new cadre will rely on correlations without prejudgments and prejudice. To be sure, subject-area experts won’t die out, but their supremacy will ebb. From now on, they must share the podium with the big-data geeks, just as princely causation must share the limelight with humble correlation.
This transforms the way we value knowledge, because we tend to think that people with deep specialization are worth more than generalists – that fortune favors depth.
About
Viktor Mayer-Schönberger is Professor of Internet Governance and Regulation at Oxford University. He is the author of Delete: The Virtue of Forgetting in the Digital Age.
Yet expertise is like exactitude: appropriate for a small-data world where one never has enough information, or the right information, and thus has to rely on intuition and experience to guide one’s way. In such a world, experience plays a critical role, since it is the long accumulation of latent knowledge – knowledge that one can’t transmit easily or learn from a book, or perhaps even be consciously aware of – that enables one to make smarter decisions.
But when you are stuffed silly with data, you can tap that instead, and to greater effect. Thus those who can analyze big data may see past the superstitions and conventional thinking not because they’re smarter, but because they have the data. (And being outsiders, they are impartial about squabbles within the field that may narrow an expert’s vision to whichever side of a squabble she’s on.) This suggests that what it takes for an employee to be valuable to a company changes. What you need to know changes, whom you need to know changes, and so does what you need to study to prepare for professional life.
Harnessing data is no guarantee of business success but shows what is possible.
The shift to data-driven decisions is profound. Most people base their decisions on a combination of facts and reflection, plus a heavy dose of guesswork. “A riot of subjective visions – feelings in the solar plexus,” in the poet W. H. Auden’s memorable words. Thomas Davenport, a business professor at Babson College in Massachusetts and the author of numerous books on analytics, calls it “the golden gut.” Executives are just sure of themselves from gut instinct, so they go with that. But this is starting to change as managerial decisions are made or at least confirmed by predictive modeling and big-data analysis.
As big data transforms our lives – optimizing, improving, making more efficient, and capturing benefits – what role is left for intuition, faith, uncertainty, and originality?
Big data is not an ice-cold world of algorithms and automatons.
Brilliance doesn’t depend on data. Steve Jobs may have continually improved the Mac laptop over years on the basis of field reports, but he used his intuition, not data, to launch the iPod, iPhone, and iPad. He relied on his sixth sense. “It isn’t the consumers’ job to know what they want,” he famously said, when telling a reporter that Apple did no market research before releasing the iPad.
Big data is not an ice-cold world of algorithms and automatons. What is greatest about human beings is precisely what the algorithms and silicon chips don’t reveal, what they can’t reveal because it can’t be captured in data. It is not the “what is,” but the “what is not”: the empty space, the cracks in the sidewalk, the unspoken and the not-yet-thought. There is an essential role for people, with all our foibles, misperceptions and mistakes, since these traits walk hand in hand with human creativity, instinct, and genius.
The same messy mental processes that lead to our occasional humiliation or wrongheadedness also give rise to successes and stumbling upon our greatness. This suggests that, just as we’re learning to embrace messy data because it serves a larger purpose, we ought to welcome the inexactitude that is part of what it means to be human. After all, messiness is an essential property of both the world and our minds; in both cases, we only benefit by accepting it and applying it.
Excerpted from Big Data: A Revolution That Will Transform How We Live, Work, and Think by Viktor Mayer-Schönberger, Kenneth Cukier. Copyright 2013 by Viktor Mayer-Schönberger and Kenneth Cukier. Reprinted by permission of Houghton Mifflin Harcourt Publishing Company. All rights reserved.
Homepage image: Sony Pictures
Editor: Sonal Chokshi @smc90
