PCMag editors select and review products independently. If you buy through affiliate links, we may earn commissions, which help support our testing.

Why Tech Companies Are Using Humans to Help AI

Called the "Wizard of Oz" technique or pseudo-AI, the practice of silently using humans to make up for the shortcomings of AI algorithms sheds light on some of the deepest challenges that the artificial intelligence industry faces.

"Andrew Ingram" is a digital assistant that scans your emails, gives scheduling ideas for the meetings and appointments you discuss with your coworkers, sets up tasks, and sends invites to the relevant parties with very little assistance. It uses the advanced artificial-intelligence capabilities of X.ai, a New York–based startup that specializes in developing AI assistants. The problems it solves can save a lot of time and frustration for people (like me) who have a messy schedule.

OpinionsBut according to a Wired story published in May, the intelligence behind Andrew Ingram is not totally artificial. It's backed by a group of 40 Filipinos in a highly secured building on the outskirts of Manila who monitor the AI's behavior and take over whenever the assistant runs into a case it can't handle.

While the idea that your emails are being scanned by real people might sound creepy, it has become a common practice among many companies that provide AI services to their customers. A recent article in The Wall Street Journal exposed several firms that let their employees access and read customer emails to build new features and train their AI on cases it hasn't seen before.

Called the "Wizard of Oz" technique or pseudo-AI, the practice of silently using humans to make up for the shortcomings of AI algorithms sheds light on some of the deepest challenges that the AI industry faces.

AI Isn't Ready for Broad Problems

Behind most AI innovations in recent years are deep-learning algorithms and neural networks. Deep-neural networks are very efficient at classifying information. In many cases, such as voice and face recognition or identifying cancer in MRI and CT scans, they can outperform humans.

But that doesn't mean deep learning and neural networks can accomplish any task that humans can.

"Deep learning is allowing us to solve the perception problem. This is a big deal because perception has limited AI since its inception over 60 years ago," says Jonathan Mugan, cofounder and CEO of DeepGrammar. "Solving the perception problem has finally made AI useful for things like voice recognition and robotics."

However, Mugan notes, perception is not the only problem. Deep learning struggles where commonsense reasoning and understanding is involved.

"Deep learning does not help us with this problem," he says. "We have made some progress in NLP (natural language processing) by treating language as a perception problem; i.e., converting words and sentences into vectors. This has allowed us to better represent text for classification and machine translation (when there is a lot of data), but it doesn't help with commonsense reasoning. This is why chatbots have largely failed."

One of the main problems that all deep learning applications face is that of collecting the right data to train their AI models. The effort and data that goes into training a neural network to perform a task depends on how broad the problem space is and what level of accuracy is required.

For instance, an image classification application such as the Not Hotdog app from HBO's Silicon Valley does a very narrow and specific task: It tells you whether your smartphone's camera is showing a hotdog or not. With enough hotdog images, the app's AI can perform its very important function with a high level of accuracy. And even if it makes a mistake every once in a while, it won't hurt anyone.

But other AI applications, such as the one X.ai is building, are tackling much broader problems, which means they require a lot of quality examples. Also, their tolerance for errors is much lower. There's a stark difference between mistaking a cucumber for a hotdog and scheduling an important business meeting at a wrong time.

Unfortunately, quality data is not a commodity that all companies possess.

"The rule of thumb is that the more general a problem an AI is trying to address, the more edge cases or unusual behaviors that can occur. This inevitably means you need vastly more training examples to cover everything," says Dr. Steve Marsh, CTO at Geospock. "Startups don't generally have access to huge amounts of training data, so the models they can feasibly build will be very niche and brittle ones, which don't usually live up to their expectations."

Such wealth of information is in the possession only of large companies such as Facebook and Google, which have been collecting the data of billions of users for years. Smaller companies have to pay large sums to obtain or create training data, and that delays their application launches. The alternative is to launch anyway and start training their AI on the fly, using human trainers and live customer data and hoping that eventually, the AI will become less reliant on humans.

For instance, Edison Software, a California-based company that develops apps for managing emails, had its employees read the emails of its clients to develop a "smart reply" feature because they didn't have enough data to train the algorithm, the company's CEO told The Wall Street Journal. Creating smart replies is a broad and challenging task. Even Google, which has access to the emails of billions of users, provides smart replies for very narrow cases.

But using humans to train AI with live user data is not limited to smaller companies.

In 2015, Facebook launched M, an AI chatbot that could understand and respond to different nuances of conversations and accomplish many tasks. Facebook made M available to a limited number of users in California and set up a staff of human operators who would monitor the AI's performance and intervene to correct it when it couldn't understand a user request. The original plan was for the human operators to help teach the assistant to respond to edge cases it hadn't seen before. Over time, M would be able to operate without the help of humans.

An Unachievable Goal?

It's not clear how long it will take for Edison Software, X.ai and other companies that have launched human-in-the-loop systems to make their AI fully automated. There's also doubt if current trends of AI can ever reach the point of engaging in broader domains.

In 2018, Facebook shut down M without every deploying it officially. The company didn't share details, but it's clear that creating a chatbot that can engage in broad conversations is very difficult. And making M available to all of Facebook's two billion users without first making it fully capable of automatically responding to all sorts of conversations would have required the social media giant to hire a huge staff of humans to fill M's gaps.

DeepGrammar's Mugan believes that we will eventually be able to create AI that can solve commonsense reasoning, what others classify as general AI. But it won't happen anytime soon. "There are currently no methods on the horizon that will enable a computer to understand what a small child knows," Mugan says. "Without this basic understanding, computers won't be able to do many tasks well 100 percent of the time."

To put that into perspective, experts at OpenAI recently developed Dactyl, a robotic hand that could handle objects. This is a task that any human child learns to perform subconsciously at an early age. But it took Dactyl 6,144 CPUs and 8 GPUs and about one hundred years' worth of experience to develop the same skills. While it is a fascinating achievement, it also highlights the stark differences between narrow AI and the way the human brain works.

"We are a very long way from having Artificial General Intelligence, and quite likely, AGI will be the combination and coordination of many different types of narrow or application-specific AI's," Marsh says. "I do think there is a tendency to overhype the capabilities of AI at the moment, but I also see there is enormous value in just taking the initial first-steps and implementing traditional Machine Learning models."

Is Another AI Winter Looming?

In 1984, the American Association of Artificial Intelligence (later renamed to Association for the Advancement of Artificial Intelligence) warned the business community that hype and enthusiasm surrounding AI would eventually lead to disappointment. Soon after, investment and interest in AI collapsed, leading to an era better known as the "AI winter."

Since the early 2010s, interest and investment in the field has been increasing again. Some experts fear that if AI applications underperform and fail to meet expectations, another AI winter will ensue. But the experts we spoke to believe that AI has already become too integrated in our lives to retrace its steps.

"I don't think we are in danger of an AI winter like the ones before because AI is now delivering real value, not just hypothetical value," Mugan says. "However, if we continue to tell the general public that computers are smart like humans, we do risk a backlash. We won't go back to not using deep learning for perception, but the term 'AI' could be sullied, and we would have to call it something else."

What's for sure is that at the very least, an era of disillusionment stands before us. We are about to learn the extent to which we can trust current blends of AI in different fields.

"What I expect to see is that some companies are pleasantly surprised by how quickly they can provide an AI for a previously manual and expensive service, and that other companies are going to find that it takes longer than they expected to collect enough data to become financially viable," says James Bergstra, cofounder and head of research at Kindred.ai. "If there are too many of the latter and not enough of the former, it might trigger another AI winter among investors."

Geospock's Marsh predicts that while funding will not subside, there will be some adjustments to its dynamics. As investors realize that true expertise is rare and only those with access to data to train the models will be differential in the industry, there will be a big consolidation in the market and much fewer startups will get funding.

"For many AI startups without a niche market application or vast amounts of data: winter is coming," Marsh concludes.

About Ben Dickson