Andrew McLaughlin is, like, California-enthusiastic. Like, we’re-totally-going-to-do-this-guys enthusiastic. The 43 year-old CEO of the recently re-imagined Digg is a believer in the way the Internet can make the world better. He grins a lot, gesticulates madly when he wants to get across an idea. Sometimes you feel like he’s trying to try to talk you into joining something. He’s like that today.
It’s a gorgeous sunny afternoon in early May and McLaughlin is walking me along the High Line in New York City, just a few blocks from the offices that house Digg and its parent company Betaworks. I’d never seen the High Line before, and McLaughlin was clearly thrilled to show it off. The High Line is where the city’s industrial past becomes something entirely modern and, at least in a boutique way, natural. A strip where nature is artfully grafted onto an old abandoned rail line, butting up against avant garde architecture, to create something beautiful; at once entirely new and constantly changing, with passing of the seasons and minute by minute as we walked through it. Theme alert: You are about to read a story about the history and rebirth of a relic of a much more recent Industrial Revolution.
"I would love to be able to configure the Digg button to just do what I want," McLaughlin says. "We’ve got to rejuvenate that whole infrastructure. Digg buttons have remained remarkably resilient around the Internet. But to rejuvenate that we have to make the verb meaningful again."
McLaughlin is talking about the future of Digg Reader, the project he and his small team of fifteen have been working on for the past month. Right now it’s just a mess of code, Keynote slides, and shit on a whiteboard. They need to turn it into a real product, one to take the place of Google Reader, which shuts down on July 1. They have less than 60 days. Simultaneously, the same team of five engineers is working to integrate another product–Instapaper–that they’ve just purchased. None of this is top secret, the opposite in fact. Digg publicly promised the world to have a replacement ready in time. They had to move fast. And when you move fast, things get fucked up.
Look, the Internet is made of fast. You go fast or you die. But lost in the Clouds of bullshit and hype there's this true thing: The internet is a technology that can connect us instantaneously to all sorts of information. That instant access lets us learn and connect and transact in entirely new ways. It’s what drives everything online–from I need to know about the Peloponnesian War right now to who is nearby that will take a couple of bucks for a spot in their back seat, sharing economy, #YOLO. It’s just impossibly fast. Even so, few things move faster than they do at the new Digg. This is the team who, in just six weeks, took a dying brand that collapsed under the weight of its own spam and made it something vibrant and vital: a place you wanted to go.
So in April, when Google announced it was shutting down Google Reader on July 1, it was almost unsurprising that Digg replied–that same day–We’ve got this.
This is the story of how a tiny team took 90 days to pull off the impossible.
John Borthwick (left) and Andrew McLaughlin in the shared kitchen space at Betaworks. Photo: Jon Snyder/Wired
That Digg would even attempt to build a Google Reader replacement speaks to the culture of its parent company, Betaworks. Betaworks is tough to describe. It starts, spins off, buys, and operates companies, mostly in media. When it comes to online publishing, Betaworks has its stakes in nearly every aspect of the business: creation, curation, authoring, measurement, discovery, sharing, saving, even gaming. As Betaworks CEO John Borthwick describes his company, he is putting together a puzzle.
You may not be familiar with Betaworks, but you use its products every day. Chartbeat, which measures the live audience on a web page at any one time–is looking at you right now. You may have arrived here via another Betaworks product: Bit.ly, the URL shortener that many media companies use not just for its ability to truncate links, but for the insights it delivers about how those links are shared. It’s possible that Bit.ly link was sent via SocialFlow, an algorithm-driven social media management program that helps brands know when and what to send to Twitter and Facebook.
Or maybe you saved this story to read later, and are now doing so on Instapaper, Betaworks’ recently purchased reading application that’s being rapidly integrated into Digg. Would you rather tell a story of your own? There’s Tapestry, Betaworks’ self-publishing app, and Editorially, a new document manager in which the company has a stake. And in case I’ve bored you away from this site, Betaworks invests in Buzzfeed, Branch, and Tumblr. Surely you can find something to read between them. Or maybe you’d prefer a game of Dots, the iOS Game of the Moment, made by–yup!–Betaworks.
All of these pieces and companies are somewhat related; they share people and data and many of the companies even work in the same studio. "The studio function is that there may be people dedicated to News.me," says Borthwick, "but if something spins up that looks related we can easily send them to that."
This means Betaworks can, say, buy Instapaper, and integrate it into Digg. Or, even better, buy the Digg.com domain and use it for one of its existing products. Which is pretty much exactly what happened.
Remember Digg? You probably used it sometime in the mid 2000s and then, unless you are a spammer, forgot about it. Once the model of Web 2.0, it eventually became an insufferable repository of internet detritus. Last summer, Borthwick discovered Digg was on the auction block, and within a week put together a handshake deal for Betaworks to buy its intellectual property. Most of it was junk. They completely abandoned the codebase, which now sits unused and unloved on a Github repository.
"We bought Digg, but it was really a carcass," Borthwick explained. "We would have had a huge legacy to deal with technically if we'd adopted the old architecture."
But there was something valuable in that rotting corpse: the Digg brand. People still associated Digg with news. Borthwick decided to keep the iconic logo, the Digg thumbs up symbol, and the domain name and use it all for another Betaworks product that had stalled out: News.me.
News.me began as a joint venture with the The New York Times’ R&D lab, run by Digg’s current CTO, Michael Young. (The Times remains an investor.) It would basically dig through social media, find things that your friends and their friends were sharing, and then show you the most interesting links from your extended network. It worked quite well, but it never really caught on. The people who used it loved it–but not enough people used it.
The idea was to take the core technology behind News.me’s tech for finding trending links, and flip it to work as a global rather than personalized tool. So instead of finding stories that were interesting to you, like News.Me, the new Digg would find stories that were interesting to everyone. People would go to Digg.com and once again find something great to read. Thanks to years and years of Digg’s media dominance and, well, all that search engine spam, the new Digg would have instant traffic.
Just as the deal was being finalized, word got out that Betaworks had bought Digg and was doing something new with it. Instead of doubling down on secrecy, Borthwick chose to go public, committing to re-launch in just six weeks. The deadline was a little bit of a PR stunt, but a fairly solid bet against fawning media attention. Amazingly, they did it–and the press was suitably glowing.
The new Digg was fantastic. It consistently found interesting stories so quickly that even serious media junkies could always find something new and different. But as summer turned into fall and then winter, the site’s momentum slowed. And so in January of this year, Borthwick asked one of Betaworks’ partners, Andrew McLaughlin, to run it full time.
Andrew McLaughlin landed at Betaworks from Tumblr that previous Fall, first as an entrepreneur in residence, and then as a partner in the LLC. But he’s had long experience solving big, complex problems. He’s the former Vice-CTO of the United States of America. For many years before that, he ran global public policy at Google. He’d helped get ICANN off the ground in the late 90s. During a professorship at Harvard, he’d travelled all over Africa helping various nations get online. In short: Dude has bona fides.
When he took over Digg, it was a bit adrift. There was talk about expanding into some sort of social commenting system (Branch, which does just that had recently moved out of the Betaworks studio.) McLaughlin wanted to tighten up, figure out what Digg did well, and make it better.
"Social is so hard, like, everyone who runs a commenting system ends up killing themselves or shooting up a post office," he explained. "I wanted it to be really, really useful and focus on the emotional experience that Digg can bring, and then back into social."
One way to do that might be via a news reader that lets people share what they were reading. In fact, it was such a great plan, he’d already tried to buy Google Reader. Kind of.
The Digg motto, sprinkled throughout the office, provides lighthearted motivation for the team. Photo: Jon Snyder/Wired
DIGG STARTS UP
McLaughlin saw a blog post in the Fall of 2012 speculating that Google Reader, choked of resources, was shutting down. He sent a teasing note to a friend at Google offering to "take it off their hands." To his surprise, he got a serious reply. Google, his friend replied, had concluded that it couldn’t sell the name, user data, or code base (which would only run on their servers) and so there was nothing to actually buy.
The following February, McLaughlin, now full-time at Digg, bumped into this same pal at a TED conference. The friend warned him to act fast if he really did want to develop a Reader. "He said ‘I’m not telling you anything, but we’re not going to keep this thing around forever and maybe you want to have something ready by the end of the year."
But instead of year’s end Google announced plans to shutter Google Reader on July 1. That same night, Digg put up a blog post announcing that it was going to build a replacement. The Internet went crazy.
The idea of Digg building a Reader replacement just resonated. The revamped Digg.com was already popular, especially in news and developer circles. It had a reputation for scrumptious headlines and kickers, courtesy of editorial director David Weiner, a HuffPo alum. Its tech team, led by CTO Michael Young had already shown serious backend chops, which meant people didn’t doubt its ability to pull off building a reader. The same minimalist sensibility that design director Justin Van Slembrouck had given the front page of Digg would translate well to the new project, and, hell: Its GM Jake Levine might even be able to figure out a way to monetize it in ways Google never had.
Because, ultimately, this is about money. Betaworks wasn’t throwing resources at a Digg Reader for altruistic reasons. The plan was to develop something with a mix of free and paid features. Maybe they’d charge a dollar for the iOS app or for tracking a large number of feeds; maybe the ability to search feeds would command a premium.
"I think we have a fantastic opportunity to be selling from day one," McLaughlin predicted during an all-hands meeting in late April. "We’ll have something for free, something for pay. What we want are users who care enough about it to pay as a base, and then to build on top of that."
Andrew McLaughlin (left) and Michael Young. Photo: Jon Snyder/Wired
A Digg Reader would also make Digg itself better. The core Digg experience is one of discovery: It constantly has to be showing you something new to work. That means it has to find stories people will care about very quickly. Right now, it does that with algorithms that analyze new links trending on social networks. To get even faster, it would need to find stories before they trend on Twitter or explode across Facebook. This is where it gets neat: If Digg had its own news reader, it could immediately identify which stories people were actually reading—not just what they click on. Digg buttons and sharing icons built into reader amplify that signal, it lets them know, immediately, that something is getting attention.
In fact, it could be a perfect fit with the rest of the Betaworks puzzle–Chartbeat could point out stories that are being read all the way through; Bit.ly could give insight into what people are linking to; and Instapaper could even show which stories people may want to read but don’t have time to, right now. Its own reader would give Betaworks a way bigger piece of the fast. But the only sure way to grab that fast was to tie its fate to the exodus of passionate Google Reader fans.
McLaughlin knows the risks of that tie-up. Those same fans who were so excited to have a new reader would turn on them in an instant if they rolled out a shitty product. "It’s like vapor good will for vaporware; we’ve benefitted by the fact that we’re kind of underdogs," he explained. "All of it could vanish."
One of many design mock-ups the team was working up. Image: Courtesy Digg
And they were already behind. Feedly, another excellent reader, was also racing to replace Google Reader. It already had four million users the day Google announced Reader’s shutdown. (The service has added another 8 million since.)
It was time to get to work. First they needed to figure out how to actually get people into a Digg Reader, whatever that would be.
"One thing I've noticed playing around with all these other readers is that they're mostly pretty crappy onboarding experiences," McLaughlin said at a meeting in early April. They needed to make onboarding–just the process of getting your feeds out of Google and into Digg–super easy.
This was exactly the kind of unsexy, necessary work that would take up much of the next three months.
The Digg team has a weekly standing meeting every Monday, where each unit–design, editorial, social, tech, business–checks in and reports what it is up to in the week ahead. Photo: Jon Snyder/Wired
"I certainly admire their ambition; it's a huge undertaking. It's going to be incredibly challenging to execute well," says Chris Wetherell. Many years ago, Wetherell built one of the most beloved, and useful products ever to be killed: Google Reader. Talking about what the Digg team faced, he shuddered, and let out a long sigh. "I think I'm having a sympathetic panic attack for them right now."
Feed reading is extremely, extremely technically complicated. For starters, there’s just a lot of stuff out there to grab and aggregate. From blogs, to mainstream publications like The New York Times, to calendar feeds and all manner of personalized services. Digg has to go get all that stuff and then bring it back to you—fast.
"The size of the corpus is incredibly challenging," explains Wetherell. "The internet is huge, but it’s a bunch of pages. The potential size of feed-reading data storage is many times larger than the web." One site, for example, may have many feeds, some based on search queries or tags, all of which have to be aggregated on Digg’s server.
"The costs associated with storing all the information necessary and delivering a world class feed reader are significant," Wetherell says. "Hopefully they have some money in the bank."
Digg, in all its glory. Photo: Jon Snyder/Wired
Here’s how that might play out in a typical reading experience. Let’s say you want to subscribe to all the photos from your Flickr contacts tagged "food." That’s a unique subset of photos that Digg has to grab from an API call. Every time you hit Digg Reader, it has to pull this data set, just for you. That kind of computational power gets very big, very quickly as you throw more and more users into the mix. And its just one task among many that a reader has to pull off.
When someone loads Digg Reader, the Digg server has to go to all the feeds this person is subscribed to, find all the newest stories that have appeared since the user’s last visit, and then assemble them in reverse chronological order. There are two ways to deliver this stuff, Digg can wait for someone to request it (pull) or send it to them in a continuously updated fashion (push). The latter is faster, but it’s also far more server-intensive. (For now, Digg is going with pull-only, and eventually wants to go to a mix of both.)
Meanwhile, the Reader is doing a lot of math. Before it can send those stories, the reader has to count which stories you’ve read and which you haven’t and display that information as well. Because Digg wants you to take action on stories (or at least be able to take action) it also has to track some associated data with each story. Have you shared it on Twitter or Facebook? Did you Digg it? Did you send it to Pocket or Instapaper to read later? If you have, it needs to display the right icon. Oh. And maybe it should go without saying, but, all this stuff is unique to you. So it has to do this again and again and again for every single user, for every single story.
A sketch of how Digg Reader works.
Image: Courtesy Michael Young
This is just the top level stuff the Digg team had to figure out in three months. There’s all manner of tedium involved as well: numerous little decisions about the way each user interaction will be handled, for example. Little things, like API caching that most people never think of when they consider "simply" cloning an existing product. And all of this has to happen very, very quickly. For an RSS reader to work, it has to be more or less instant.
"There are two sides to fast," explains Digg’s CTO Michael Young, referring to the front-facing software and the backend full of data. "A reader’s got to load fast, and it's got to be timely so that we get these stories as soon as they're coming off the publishing system." Both require a lot of work, but timeliness is the real killer hiding in the closet.
An early design mock-up. Image: Courtesy Digg
"RSS is painful. Take the Wired RSS; I have to check it every so often." (At this point, Young begins to impersonate a computer pinging a server.) "Is there a new story? Is there a new story? Is there a new story? If it's more frequent than, say, every 15 minutes, some publisher sites will block me." More on that in a minute.
It’s a fundamentally broken system. Subscribers want to see new stories in their feed readers as soon as they appear online. But unless publishers have implemented a method to push updates, Digg Reader won’t see them except when it crawls feeds, which, in its current form, it only does every 15 minutes. Some sites don’t publish but once every few weeks, and pinging them every fifteen minutes is overkill. For others that spew news or serve up, say, real-time traffic updates, it is far too slow.
One of many design mock-ups for tablet reading. Image: Courtesy Digg
Imagine this scenario: Digg crawls Wired’s feeds at 3:00, Wired publishes a new story at 3:01, and I load Digg Reader at 3:14. I’m not going to see that news–even though it’s relatively ancient by Web standards at that point.
"That's one of the biggest challenges that keeps me shaking my head in sympathetic stress thinking about tackling that problem outside of Google," says Wetherell. "One of the reasons for Google's Reader's success is that it got to piggyback off of the world's best crawling system."
Everyone publishing on the internet wants to be in Google because Google means traffic; and traffic means money. Publishers actively want Google’s robots to access their sites as frequently as possible. But they don’t necessarily want sixteen dozen random feed readers pinging their systems again and again and again. Hit them too often, and they may punish you, blocking your robots. Lose your robots, and you lose your readers.
When I visited Digg in early May, the whiteboard was a sea of action items. Littered among them were in-jokes like "NSA Wiretap" (this turned out to be prescient) and "Windows Phone app" (when I asked about this they just laughed).
Yet by May 31, much of those action items were already accomplished, and the team was ready to move Reader off the staging server (notouching.me, an Arrested Development reference) to Digg.com. Almost instantly, the entire site went down. Young was "two beers in" when he realized that every visitor to Digg was being greeted with a 503 error. Oh shit, oh shit, oh shit!
Turns out, they just needed to reboot a router.
Jake Levine and Andrew McLaughlin look over Rob Haining's shoulder, while Justin Van Slembrouck attempts to focus. Photo: Jon Snyder/Wired
At an all-hands meeting on June 3rd, the tone was celebratory. Rob Haining, Digg’s iOS developer, had just submitted the app to Apple for approval. Sure, the backend driving it wouldn’t be ready for another few weeks, but this was a big milestone. And moreover, the race was nearing its end. They were really going to do it, even if there was still lots to be done.
"The goal is still to have an Android app out there by launch date," said McLaughlin, "whether or not that's going to happen I'll keep you guys posted."
It didn’t. Android had been a persistent problem throughout the development cycle, but it was just one of many things that didn’t make the final cut. Another is search. Searching feeds is one of Google Reader’s most powerful features, but it’s also a bitch to implement. None of the other major news readers have a search function, and Digg won’t either–at least not at launch.
"In terms of technical execution, the number one incredibly difficult feature is search," explained Wetherell. "That is something Google had that would be very difficult at other companies."
"Search still has to get added in," says McLaughlin. "That's really just a matter of manpower on the backend. We know how to do search. We'll get it. That one is pretty expensive, you have to run a lot of operations and store a lot of data."
Google Reader users won’t be able to import their tags either. "If you get Google Takeout they don't even give you your tags," Young explained at this week’s all hands meeting. (Takeout is Google's system that lets users download their data from various Google products.) There wasn’t a way around it, either. "We should just make it clear to people."
It is, however, a hell of a start. Launching was all about getting a product out the door, and Digg Reader meets pretty much all the goals the team set for itself. It’s got a slick minimalist design that, yes, looks very much like Google Reader (and very much like Feedly, for that matter). It has built-in sharing and saving features. The Digg button will help find stories for the site’s front page. The iOS app is fantastic (it even has a car mode for podcasts). It has read counts, and they work, which sounds easy to pull off but requires lots of complex things happening in real time on the back end. (Which is why Google Reader’s unread count maxxed out at 1000+).
The final product.
All that remains to be seen is if it is fast, and if it can scale. Which are two huge questions that will be somewhat answered this weekend when Digg opens its doors to the 18,000 people who answered the "what do you want in our Reader" survey Digg sent out in April.
Impressive as this three-month sprint is, it’s still just a foundation. The really interesting stuff is yet to come. McLaughlin and team’s vision isn’t just to replace Reader, it’s to one-up it for a new era. The biggest problem with RSS has always been THERE IS TOO GODDAMN MUCH RSS OH MY GOD HELP ME I’M DROWNING. Digg wants to use everything at its disposal–from Betaworks tools like Chartbeat and Bit.ly to its own trackers to external social signals to solve that, to find the things that really are important to you.
The Digg team certainly has a shot at cracking that nut; their progress so far is astonishing, and their relentless enthusiasm makes you almost believe they can accomplish anything. But they also might melt the wax on their wings, come crashing down into the hard crust of the real world, and lose an astonishing amount of money. It almost doesn’t matter. The engine of the Internet has always been reckless ambition, and right now Betaworks is a large part of what’s moving us foward: inspiring us, competing with us, making us jealous and dismissive and goading our creative monsters to come out and play.
