Algorithms Have Already Gone Rogue

In his new book, Tim O'Reilly identifies the first case of AI run amok—and it's not what you think.
Image may contain Audience Human Crowd Person Speech Lecture Finger and Tim O'Reilly
Kelly Sullivan/Getty Images

For more than two decades, Tim O’Reilly has been the conscience of the tech industry. Originally a publisher of technical manuals, he was among the first to perceive both the societal and commercial value of the internet—and as he transformed his business, he drew upon his education in the classics to apply a moral yardstick to what was happening in tech. He has been a champion of open-source, open-government, and, well, just about everything else that begins with “open.”

His new book WTF: What’s the Future and Why It’s Up to Us seizes on this singular moment in history, in which just about everything makes us say “WTF?”, invoking a word that isn’t “future.” Ever the optimist, O’Reilly celebrates technology’s ability to create magic—but he doesn’t shirk from its dangerous consequences. I got to know Tim when writing a profile of him in 2005, and have never been bored by a conversation. This one touches on the effects of Uber’s behavior and misbehavior, why capitalism is like a rogue AI, and whether Jeff Bezos might be worth voting for in the next election.

Steven Levy: Your book appears at a time when many people who once had good feelings towards technology are now questioning it. Would you defend it?

Tim O’Reilly: I like the title WTF because it can be an expression of amazement and delight or an expression of amazement and dismay. Tech is bringing us both. It has enhanced productivity and made us all richer. I don’t think I would like to roll back the clock.

Not that rolling it back is an option.

No, but it’s important for us to realize that technology is not just about efficiency. It’s about taking these new capabilities that we have and doing more with them. When you do that, you actually increase employment. As people came off the farm, we didn’t end up with a vast leisure class while two percent of people were feeding slop to animals. We ended up creating new kinds of employment, and we used that productivity actually to enhance the quality and the quantity of food. Why should it be different in this era of cognitive enhancement? Uber and Lyft are telling us that things we used to think of as being in the purely digital realm, in the realm of media, whatever, are coming to the real world. So that’s the first wake up call for society. Secondly, we’re seeing a new kind of interaction between people and algorithmic systems. Third, they represent a new kind of marketplaces based on platforms [in this case, they exist because the of the platform of smartphones—and then they can become platforms of their own, as new services, like food delivery, are added in addition to transit]. This marketplace works because people are being augmented with new cognitive superpowers. For example, because of GPS and mapping apps, Uber and Lyft drivers don’t need a lot of training.

Agreed. But when the curtain rolls back we see that those superpowers have consequences: Those algorithms have bias built in.

That’s absolutely right. But I’m optimistic because we’re having a conversation about biased algorithms. We had plenty of bias before but we couldn’t see it. We can’t see, for example, that the algorithms that manage the workers at McDonald’s or The Gap are optimized toward not giving people full-time work so they don’t have to pay benefits. All that was invisible. It wasn’t until we really started seeing the tech-infused algorithms that people started being critical.

In WTF you talk about a specific out-of-control algorithm: the capitalist impulse to maximize profits regardless of societal consequences. The way you describe is reminds me of Nick Bostrom’s scenario of an AI machine devoted to making paper clips—because that’s its sole mission, it winds up eating up all the materials in the world and even killing those who would turn it off. Corporations whose sole justification is shareholder value seem to be working on a similarly destructive algorithm.

Yes, financial markets are the first rogue AI.

How do you roll back that particular AI?

I try to show [earlier cases of] how humans tried to manage their algorithms, by talking about [how Google improved] search quality. Google had some pretty bad patches where the spammers really had the upper hand, and they addressed it.

And that can be done to fix capitalism’s rogue AI?

Somebody planted the idea that shareholder value was the right algorithm, the right thing to be optimizing for. But this wasn’t the way companies acted before. We can plant a different idea. That’s what this political process is about.

Speaking of politics, it seems like another runaway algorithm has led us to a government controlled by people who don’t represent majority views.

I look at it through the long arc of history. You look at the long slow decline of the Roman Empire and see so many analogies—the turning away from literacy and science, the outsourcing of core functions of government to mercenaries effectively. We could go through some real shit before we turn around. We might not turn around at all. But I take hope from something that Tim Urban in Wait But Why calls “the human colossus.” He has this fabulous description of how Elon Musk moves this human colossus in a new direction—to show that it’s possible to go into space, to show that it’s possible to build a brain-machine interface—and then everybody else will follow along. The human colossus I’m most heartened by is the post-World War II period. We learned a lesson from the incredible convulsions after World War I where there was vast dislocation, as we punished the losers of the war. So after World War II they rebuilt Europe, and they invested in the returning veterans with the GI Bill.

As we learn from tech, though, algorithms need continual improvement. You don’t just set them in motion and leave them forever. The strategies put in place after World War II that worked for this period of 30 years have stopped working so well, so we came up with something else [which happened to create income inequality]. There’s a sense that things going wrong will lead to new strategies. And now that Trump has broken the Overton Window—

What’s that?

It’s this idea [named for the late think tank leader Joseph Overton] that there’s a certain set of things that are considered acceptable in public policy debate, and you just can’t go outside that window. And Trump has just done everything unthinkable. Because all bets are off, we are not necessarily going back to the old, tired solutions. I think it’s possible that we’ll shrug off this madness, and we will come back to saying we really have to invest in people, we really have to build a better economy for everyone. In China, they’re already doing that. China has recognized that its vast population is a possible powder keg and it has to take care of its people. That’s something we have not done. We’ve just been pushing down the ordinary people. China is also being more aggressive than any other country in rising to the challenge of climate change. So there’s two possibilities—we’re going to wake up and start acting the same way, or China will lead the world.

Reading your book I think I know who you’d like for our next president: Jeff Bezos. The book is full of Bezos love.

Well. Jeff and Elon [Musk] are probably the two entrepreneurs I admire most.

You can think of the book as an apology to Jeff. As a publisher, I originally bought the usual story, that Amazon would go the way of Wal-Mart—the more dominant it got, the more it would extract value for itself, squeezing down its suppliers. Jeff is a ruthless competitor, no question, but while Amazon has done a chunk of that, it has spent so much time trying to do more. I’m not sure that Jeff would make a great president, but he might.

You’d vote for him, wouldn’t you?

It would depend who he was running against but, yeah, I probably would.

You also praise Uber in your book. Do you think it’s possible to distinguish between the value of its service and the ethics of the company?

Uber is a good metaphor for what's right and wrong in tech. Here we have this amazing new technology, which is transforming an industry and putting more people to work than worked in that industry before, creating great consumer surplus, and yet it has ridden roughshod over cities, and exploited drivers. It’s interesting that Lyft, which has been both more cooperative in general and better to drivers, is gaining share. That indicates there’s a competitive advantage in doing it right, and you can only go so far being an ass.

Let’s finish by talking about AI. You seem a firm believer that it will be a boon.

AI itself will certainly not take away jobs. I recently saw a wonderful slide from Joanna Bryson, a professor from the University of Bath. It referred to human prehistory and the text said, “12 thousand years of AI,” because everything in technology is artificial intelligence. What we now call AI is just the next stage of us weaving our intelligence together into a greater whole. If you think about the internet as weaving all of us together, transmitting ideas, in some sense an AI might be the equivalent of a multi-cellular being and we’re its microbiome, as opposed to the idea that an AI will be like the golem or the Frankenstein. If that’s the case, the systems we are building today, like Google and Facebook and financial markets, are really more important than the fake ethics of worrying about some far future AI. We tend to be afraid of new technology and we tend to demonize it, but to me, you have to use it as an opportunity for introspection. Our fears ultimately should be of ourselves and other people.