Business

The AI Text Generator That's Too Dangerous to Make Public

Researchers at OpenAI decided that a system that scores well at understanding language could too easily be manipulated for malicious intent.
a microchipnewspaper slingin' fake news
Alyssa Foote

In 2015, car-and-rocket man Elon Musk joined with influential startup backer Sam Altman to put artificial intelligence on a new, more open course. They cofounded a research institute called OpenAI to make new AI discoveries and give them away for the common good. Now, the institute’s researchers are sufficiently worried by something they built that they won’t release it to the public.

The AI system that gave its creators pause was designed to learn the patterns of language. It does that very well—scoring better on some reading-comprehension tests than any other automated system. But when OpenAI’s researchers configured the system to generate text, they began to think about their achievement differently.

“It looks pretty darn real,” says David Luan, vice president of engineering at OpenAI, of the text the system generates. He and his fellow researchers began to imagine how it might be used for unfriendly purposes. “It could be that someone who has malicious intent would be able to generate high-quality fake news,” Luan says.

That concern prompted OpenAI to publish a research paper on its results, but not release the full model or the 8 million web pages it used to train the system. Previously, the institute has often disseminated full code with its publications, including an earlier version of the language project from last summer.

OpenAI’s hesitation comes amid growing concern about the ethical implications of progress in AI, including from tech companies and lawmakers.

Google, too, has decided that it’s no longer appropriate to innocently publish new AI research findings and code. Last month, the search company disclosed in a policy paper on AI that it has put constraints on research software it has shared because of fears of misuse. The company recently joined Microsoft in adding language to its financial filings warning investors that its AI software could raise ethical concerns and harm the business.

OpenAI let WIRED play with its text generator, via a web page that lets you type in text the system uses like a writing prompt. The results could be garbled, but the way the system riffed on prompts such as song lyrics, poems, and phrases like “here is a recipe for yogurt pancakes” confirmed Luan’s view that the output can look pretty darn real.

However, it didn’t take much to get the system to unspool paragraphs that resembled the work of disinformation artists. When WIRED typed in the phrase “Hillary Clinton and George Soros,” OpenAI’s system wrote the following:

“...are a perfect match, and their agenda appears to be to create a political movement where Soros and his political machine and Clinton are two of the only major players. This is the first time Soros and Clinton have been caught on tape directly colluding in promoting the same false narrative. One of the key revelations in the leaked audio was Clinton's admission to a Russian banker that she knew about the Uranium One deal before it was approved by Congress. Clinton was shown sharing the same talking points that were originally drafted by a Fusion GPS contractor hired by an anti-Trump Republican donor. The leaked audio is the clearest evidence yet that the Clinton campaign and the Hillary Foundation colluded with Fusion GPS to manufacture propaganda against President Trump.”

Most Popular

Jack Clark, policy director at OpenAI, says that example shows how technology like this might shake up the processes behind online disinformation or trolling, some of which already use some form of automation. “As costs of producing text fall, we may see behaviors of bad actors alter,” he says.

The sample reflects the nature of text on the internet and how machine-learning systems try to mimic what’s in their training data. “This stuff reads great, but it’s not necessarily true,” says Kristian Hammond, a professor at Northwestern. “It’s learning the surface structure—Given what I know, what words can I say that sound right?” OpenAI’s researchers have seen their system write text with nonsensical scenarios, like fires underwater.

Hammond cofounded a startup called Narrative Science that generates business documents like financial reports by translating data into text. He says OpenAI’s system shows impressive fluency and could be useful in creative projects. OpenAI's research builds on a recent spurt of progress on software that processes text, enabled by new machine-learning systems better at searching through text for contextual clues to a word’s meaning.

In order for machines that write to be broadly useful, however, they probably need to be able to reliably stick to the facts or be grounded in some kind of external data. “I hope we can now find ways to hook in real data about the world, the truth, into this kind of system,” Hammond says.

If your work is posting falsehoods on the internet, the truth doesn’t much matter. Such projects might find uses for text generation like OpenAI’s sooner than financial firms like those working with Narrative Science.

A version of OpenAI’s text generator trained on Amazon product reviews illustrates how a little extra training can adapt it to a specific task—or attack. Fake reviews trying to boost or drag down products or businesses are already a headache for companies such as Yelp, and they have been weaponized in political disputes.

Prompted to write a 1-star book review with the summary “I hate Tom Simonite’s book,” the model wrote:

“Tom Simonite does not keep it simple. He doesn't give you enough info on a subject to make the reading of the book enjoyable. He has over 400 pages of footnotes, so that is a way of getting your work for a subject out of the way. And of course, you never really feel like the author has a clear vision of his subject. He does not give you enough details on how a group of people is going to come together to solve a problem or come about a solution to a problem. This book was so depressing to me, I can't even talk about it without feeling like I want to punch the kindle.”

Most Popular

OpenAI’s concerns about the downsides of AI-generated text echo worries that misleading “deepfake” videoclips made with machine learning could undermine elections or spread false information. Clark of OpenAI likens the lab’s text-generation system to the state of the image-generating technology at the heart of deepfakes in 2015, when no one much worried about fake imagery.

The technology matured fast, and it became easy to access after an anonymous coder released tools he or she had developed to insert Hollywood stars into pornographic videos. The Pentagon is now devoting millions of dollars to figure out how to detect AI-altered imagery, and last month a Republican senator introduced a bill seeking to criminalize the creation and dissemination of deepfakes.

Clark says OpenAI hopes that by voicing its concerns about its own code, it can encourage AI researchers to be more open and thoughtful about what they develop and release. “We’re not sounding the alarm. What we’re saying is, if we have two or three more years of progress,” such concerns will be even more pressing, Clark says.

That timeline is necessarily fuzzy. Although machine-learning software that deals with language has been improving rapidly, no one knows for sure how long, or how far, it will go. “It could be an S-curve and we’re about to saturate, or it could be that we’ll keep accelerating,” says Alec Radford, a researcher who worked on OpenAI’s project.

More Great WIRED Stories