Just before 9 am last Thursday, an unusual speed dating scene sprang up in San Francisco. A casually dressed crowd, mostly male, milled around a gilt-edged Beaux Arts ballroom on Nob Hill. Pairs and trios formed quickly, but not in search of romance.
Ice breakers were direct: What’s your favorite programming language? Which data analysis framework are you most expert in? More delicately, conversations drifted toward rankings on Kaggle.com, a site that has turned data science into a kind of sport.
The more than 200 attendees, drawn from the site’s top echelons, formed teams for an eight-and-a-half-hour data-crunching challenge. It was part of an event called Kaggle Days, organized by Warsaw startup LogicAI to give some of the site’s devotees a place to mingle and compete offline. Entrants were given data from an anonymous auto parts maker and asked to predict bad batches in factory output. One team stood out because it openly intended to cheat: a trio of Google researchers testing artificial intelligence software called AutoML, designed to do the work of a data scientist.
Recent advances in AI have raised questions about the effect of smarter machines on jobs for humans. They have mostly focused on relatively low-status work such as driving trucks and checking out shoppers. Last week’s experiment offered a look at how AI may change other strata of the labor market, too. In AutoML, the high priests of the technology—some of the world’s most highly valued employees—direct the technology to disrupt their own work.
Ninety minutes into the contest, competitors had burrowed into the data, and favored working spots. A few secreted themselves in quiet corners of the hotel. Most bent over laptops in two windowless ballrooms well supplied with coffee, energy-rich snacks, and Ethernet connections.
In one of those rooms, Vladimir Iglovikov, one of the “grandmasters” at the top of Kaggle’s rankings, stood by to offer tips to competitors who needed help. He credits Kaggle with helping him rise from crunching data at a collection agency to working on vision systems for self-driving cars at Lyft—an example of how the site’s top performers can find their lives transformed by the skills and cachet won in competition.
Would AutoML change that? Iglovikov was doubtful that AI software could match the creativity of the world’s top data-science obsessives soon—a view shared by other grandmasters watching Thursday. But he could see automated AI being disruptive inside companies. “I can replace some of my time with a computer’s time,” he said. Companies that make scant use of data science today due to a lack of expertise or resources would have the most to benefit, he said. Software, Iglovikov noted, doesn’t require vacations, visas, or a salary.
The competitors toiled in the shadow of a leaderboard projected onto a large screen. Kagglers gauge their progress during a competition by submitting code to the site for testing, and receive a score that’s posted publicly. Final positions aren’t revealed until a contest ends, when code submissions are scored on data unseen by the competitors.
Not long after 11 am, about two hours into the contest, the AutoML team submitted its first auto-generated code—and debuted in second place on the leaderboard.
AutoML’s origins can sound like a sci-fi writing prompt or the brainchild of PhD-level slackers. About three years ago, some of the researchers Google pays handsomely to invent new AI software invented AI software to do some of their work. Their meta-level AI was soon better at some parts of their job than they were.
Much recent AI technology, like the speech recognition of a smart speaker, derives from programs called neural networks. Google’s AI prowess comes in part from its researchers creating new shapes, or architectures, for those networks, which process data in ways inspired by the neurons of human brains.
AutoML created software that could automatically generate and test new neural network architectures. Its creators found that over time this process could discover more powerful and efficient models than they could. Today, the most accurate results achieved on a standard benchmark for visual AI software, ImageNet, were achieved by neural networks designed by neural networks, not humans.
In 2018, Google’s cloud division released a commercial version of AutoML to help others create custom image-recognition software. The day before last week’s contest, the company announced that version can now handle video, and data formatted in tables.
That product is designed to attract new customers for machine learning services, which Google uses to differentiate itself from cloud market leaders Amazon and Microsoft. Kaggle serves a similar function—since Google’s cloud unit acquired the site in 2017, it has expanded features that help newcomers to machine learning share code and ideas outside of its signature competitions.
The AutoML team competing on Nob Hill used a research grade edition of the software, not the commercial version. Shortly before noon, they submitted a second set of code from their software, and it took the lead.
Quoc Le, the soft-spoken AI researcher who led the creation of AutoML, found that somewhat surprising. After testing AutoML against past Kaggle competitions, which typically take place over months, not hours, he and his team thought that finishing in the top 10 percent in the live contest would count as success. As Le sat beside the artificial lagoon in the hotel’s dimly lit tiki bar, competitors dashed in to grab boxed lunches before heading back to their laptops.
“There are a lot of parts of our work that are very tedious and I don’t want to do,” Le said, when asked about AutoML’s origins. Automating them frees him to spend time thinking about projects that could bring about more significant advances in AI, he said. Le believes people outside of AI research should see similar benefits, pointing to how chess computers have helped elevate the game, not made human chess players extinct. “Humans have lots of knowledge that I don’t think AutoML will be able to figure out,” he said. Le is thinking about creating a “Kagglebot” that routinely enters the site’s contests.
When Le returned to the ballroom where his two fellow Googlers were supervising their automated teammate, he surveyed the leaderboard. AutoML was still at the top. “So far so good,” Le said.
By 3:30 pm, a robot victory seemed assured. AutoML’s lead seemed unassailable, with the closest humans a good distance behind. Then the Google bot lost its footing. When competitors assembled at 5:30 pm to see the final scores, a hearty and relieved cheer broke out. AutoML had finished second.
Humanity’s victory came via a duo who met for the first time that morning. Erkut Aykutlug, a data scientist for Sony in Orange County, had teamed up with Mark Peng, who works from Taiwan for Minneapolis startup Exosite, which develops software for monitoring buildings and industrial equipment.
Peng, in a puffy jacket and floppy hair, credited their success in part to insight gained from building several types of models to examine the dataset. Those different perspectives helped inspire better ways to handle problems like missing data values. He was unruffled by Google’s AI software finishing close behind.
“I don’t think AutoML will replace data scientists,” Peng said. He suspects the resources needed to make AutoML practical and powerful will put it beyond the reach of all but the largest companies and projects. Google takes a different view—the company is betting that it can make AutoML both smarter and cheaper, in part through increasing the power of its in-house AI chips. When Peng reflected on the ambition of company’s project, he couldn’t help but marvel. “It’s pretty crazy,” he said.
Google’s Le remained cheerful, saying he was pleased with second place and had enjoyed the last-minute drama. Asked what was next for his research project, a determined look crossed his face. “I’m impressed by this team,” he said of the winners. “I want to ask a little bit about how they did it.”
- The defiant, restorative joy of lurking online
- Is Facebook’s ad system hard-coded for discrimination?
- Want to build a better democracy? Ask Wikipedia how
- Hacker Eva Galperin has a plan to eradicate stalkerware
- UPS drones are moving blood samples over North Carolina
- 👀 Looking for the latest gadgets? Check out our latest buying guides and best deals all year round
- 📩 Hungry for even more deep dives on your next favorite topic? Sign up for the Backchannel newsletter
