My Launch Lessons from 37 Minutes in an Amazon War Room
Product

My Launch Lessons from 37 Minutes in an Amazon War Room

Before joining Twitter, Ibrahim Bashir led the launch of the third-generation Kindle at Amazon. Here, he reflects on his lessons from that release — and shares how to prepare to make hard decisions quickly when the unexpected happens.

In the late morning of April 11, 2011, hours before its planned launch, the third-generation Kindle—the first lower-priced Kindle with Special Offers—was leaked. Moments later, 20 people in a Seattle conference room jumped into gear. Thirty-seven minutes after that, the device was officially unveiled and available for purchase, and Jeff Bezos was getting ready to sing its praises in a press interview.

How is it possible to launch a new product line in less than an hour? For starters, the Kindle team was as prepared for surprises as they were for a regularly scheduled launch. With tech watchers sniffing around for details of the next Kindle and journalists holding onto an embargoed press release, there was a very real possibility that word would get out sooner than the team intended.

Still, preparing for an accelerated launch is one thing. Realizing that your product just became the hottest gadget in town — and you don’t have so much as a Buy button to show for it? That’s quite another.

Ibrahim Bashir—then senior manager for Kindle, now director of program management and engineering at Twitter—was at the helm that day. Now, with a few years of perspective, he walks through those 37 minutes and the hard-won lessons that’ll help startups counteract any trials or turbulence on launch day.

Lesson #1: Don't Fight the Leak.

Faced with a leak, the most effective response will vary from company to company and launch to launch. In some cases, you might issue a denial; in others, an “any press is good press” approach is in order. Or if you’re, say, Apple, you’ll just completely ignore the noise and proceed with your meticulously planned launch event. Just don’t waste your time trying to plug a leak.

In this case, the Kindle team knew that they could, and should, capitalize on any early buzz, and the website traffic that came with it. “We wanted to steer into the leak,” says Bashir. “To help determine your approach, mark down the earliest possible date you’d feel okay — even if slightly uncomfortable — launching. Our Kindle sales team had determined a game-time window. If news leaked more than a week before the scheduled release, our team’s hands were tied. Launch too far before the product was available to ship, and customers would grow frustrated by the long wait. A leak after that point, though, would immediately trigger launch.”

As part of its contingency prep, the team had also determined how they would modify the master launch plan in the event of a leak. That’s lesson #2: build a plan for partial or rapid release into your launch strategy. From leaks to system outages to unexpected competitor moves, there are any number of reasons a company might need to move quickly on a big announcement.

In this case, Bashir knew exactly how to proceed with the shortest possible launch; the “leak script” even had its own column in his launch spreadsheet. “Once you get down to the bare essentials, you know that search has to work, campaigns have to look normal, pricing has to be correct, and customers have to be able to purchase and get an order confirmation.” Thanks to a series of dry runs—of both the ideal scenario and the leak version, too—he also knew exactly how long it should take.

Having a list of bare essentials and doing dry runs are critical because you’ll naturally have competing expectations around speed in your company. “The launch team always pushed for more time, because you don't want anybody in the room feeling pressure,” says Bashir. “The PR team and everybody else pushed for less time. ‘Can we do this in 15 minutes? Why can't we do it in 15?’"

Ultimately, it comes down to determining how long you need to accomplish the must-haves and achieving consensus about which items don’t have to work perfectly from the get-go. “In a leak scenario, fine, customers won't be able to write reviews for the next couple of hours. We'll live with that. Or you might see some bogus search results. We'll live with that.” After having had the conversations and done dry runs, the team arrived at a number everyone was comfortable with.

So when the leak happened and the countdown was on, everyone knew what they were working with: 45 minutes on the clock.

Lesson #3: Plan Every Detail.

The project team had been sequestered in a war room for the last couple weeks of the project, preparing for launch (and running through contingency plans in case things didn’t go as planned). That day, the member of the Comms team tasked with monitoring social media noticed a clear spike in buzz. The PR pros jumped into gear, confirming what looked increasingly clear: this leak was the real deal. It was go time.

If you’re an early-stage startup, you may be thinking that it’ll be a while before the world is banging down your door for the latest product news. But the mechanics of a major product launch—the prioritization, painstaking planning, and clearly articulated delegation—have broad applications. Maybe you need to deal with a new competitor, for example, or a website outage.

A war room mentality is not just a mindset; it’s a muscle your startup must exercise — and not just for launch.

This particular war room had been set up in the Kindle building on the Amazon campus, and every detail was thought out to reduce friction. “We needed a space that other Amazon employees could get in and out of easily, which limited us to part of the first floor,” says Bashir. “We picked the closest room to the front door. And since it was on the ground floor, there was no elevator drama. I had heard that once, there was elevator drama—you can't have that.”

No detail was left to chance. There was even a table ready to load with food and drinks, and a plan for acquiring plenty of sustenance from the closest cafeteria.

Lesson #4: Populate the Room Wisely — and Sparingly.

The people in the room had been carefully thought out, too. “There were decision-makers and people who had their fingers on buttons,” says Bashir. “Those were the only two types of people in the room.”

So how were they chosen? Bashir deferred to each department involved in the launch. “You tell those teams, ‘We're going to have this war room and these are the decisions and the types of problems we're going to face. Please elect one person to be in the room,’" says Bashir. “Sticking to one person per team is key; there isn’t time during a launch to risk disagreement. You want one voice.”

There are many factors to consider when selecting the right representatives. Here are key questions to ask:

  • Who has subject matter expertise?
  • Who has a sense of ownership?
  • Who doesn't crack under pressure?
  • Who has been through something like this before?
  • Or who do you want to have this experience?

Sometimes leaders prefer to be in the room themselves; other times, they may prefer to send team members with more technical expertise. There’s room for a mix of both types of people, but on some level, you’ll only truly know when the countdown begins. “There are people who enjoy the adrenaline, and there are people who don't,” says Bashir. “It’s really hard to figure out who is who until you've been in the war room with them.”

Equally important is who’s not in the room. Every superfluous voice increases your risk of unnecessary distraction or disagreement. There is no room for observers or “groupies”—and more often than not, that goes double for execs. “One time, a high-ranking and senior member wanted to join the war room to see how the sausage was made. However, in this case, he would not be making any decisions or pressing any buttons, just merely observing and possibly distracting,” says Bashir. “His mere presence would have no impact on the team, beyond possibly psyching them out. That can be huge when the clock is ticking, so it was best that he wasn’t present for this launch, but debriefed afterwards.”

Also not typically needed in the war room? Product managers. By the time you’re communicating major product initiatives to the public, the time for negotiating what you’re communicating is long over. “All of the passionate stuff about what feature should be on the device or not or which markets you’re building for? It's been decided. The experience you’re shepherding out the door now is the sales experience,” says Bashir.

Lesson #5: Make Yourself Heard. Literally.

With not a minute to spare, Bashir, as senior manager for the new product line, assumed his role as the “launch manager” at the center of the action. “If you've ever seen Apollo 13, the NASA room, it looked like that,” he said.

In preparation for launch, the team had thought of everything, down to the practical tools they would need. Now, Bashir calmly put on his headset microphone. “When we practiced, everyone said, ‘You’re not a loud guy. You’re going to need a mic,’” he said. “That’s why dry runs are so vital. You conceptually get that there will be energy and a din in the room, but you have to experience it. If we found out they couldn’t hear me on launch day, we may not have hit our deadline.”

With his headset on, Bashir propped up a whiteboard, which listed the key events he needed the team to remember—the milestones they absolutely couldn’t punt on. Tools such as these actually supported — and preserved — Bashir’s voice for only the most important communication.

“You can use very 101-level tools,” says Bashir. “There was the triage corner, where we hung a poster board and marker to note major issues as they arose. A screen in the middle of the room was projecting the master spreadsheet of every task that needed to happen, with major milestones highlighted. And each of those tasks included a projected completion time, so I—and anyone else in the room—could quickly assess whether we were on track to hit that 45-minute deadline.”

“You don’t want anyone in the room coming to you and saying, ‘So-and-so wants to know what's going on.’ The PR people had been through dry runs with us, so they could just look at that hot list and literally every five to ten minutes go next door and update the execs on the phone,” says Bashir. “They,d say, ‘Here's where the team is at, here's what's going on, here's where we expect to be in the next 15 minutes.’ That keeps everybody calm and informed.”

Indeed, your most valuable tool during a launch is, simply, people. Which is all the more reason to abide by lesson #6: Give every person in the room a clear role and set of responsibilities.

The goal is to ensure that you’ve filled the room with the most qualified people, so don’t give too much thought to titles or even functions. “We had a guy who is a merchandiser—whose job is to buy products that sell on Amazon for different categories—but he was an expert with catalog systems,” says Bashir. “We would pull him in for product launches and get him up to speed. He knew who to talk to and what to do if anything funky happened with our catalogs.”

Lesson #7: Structure Launch into Phases with Clear Entry and Exit Criteria.

The room was up and running and key players were at their posts. Now, it was time to move. Bashir’s launch spreadsheet had been carefully structured to move from one key task to the next in clear, quick succession: In this case, here was the punch list:

  • Put the detail page up
  • Get the buy button working
  • Get search and campaign working
  • Get order confirmation working
  • Publish the press release
  • Lift the press embargo

There was no going back on this particular launch, but you may encounter scenarios where you want or need to undo something—or to scrap a launch attempt entirely. Whatever the exigencies of your particular scenario, proper launch hygiene demands that you move neatly, step by step.

With leaks, move with rhythm. Don’t step, then skip, then leap. Even if you know where you’re going and have to change direction.

You can further streamline a phased rollout by adhering to Bashir’s lesson #8: have launch down to a series of switches. Amazon, like most other tech companies, first builds new pages or features in an invisible staging area, keeping them hidden until it’s time for the world to see. At the simplest level, the next 45 minutes would be about flipping a series of switches to “turn things on” in the prescribed order.

“You flip a switch. You see how it reacts. And then it's, ‘Go to the next one,’” says Bashir. With all eyes on the monitors around the room displaying website traffic and other key systems, his team moved methodically through their list: First catalog details went up, then they confirmed the product webpage looked normal and all URLs were directing normally. “When it’s go time, you can’t be doing rollbacks, creating editorial, copying and pasting information. Any manual effort leads to mistakes, particularly in a tense situation.”

Sure, not every company has Amazon-level infrastructure. But even the most cash-strapped startup can afford to implement some version of these switches. “There are companies out there now that will sell you A/B testing framework and gradual feature rollout. Invest in this software,” says Bashir.

Ibrahim Bashir

Not every detail, though, was hidden behind a switch. Because while efficiency is king in moments like these, one thing trumps it. And that’s lesson #9: identify your differentiators, your major selling points, and keep them under lock and key.

“It’s gamesmanship. It’s competitive advantage. If you’re a startup, price may not be the thing you’re selling, but what is? Is it the aha moment of the product reveal itself? Is it the integration? Is it the partnership? Is it some ‘wow’ feature? The thing you think will sell your product is the thing you save for the reveal.”

For the team behind Kindle with Special Offers, price was a key element of the big reveal. In preparation for launch, they had built catalog pages for the device and its accessories, inputting details according to how critical each product was deemed. “For example, for the Kindle covers, we filled in all the colors, uploaded the images, put in the price, everything,” says Bashir. “It's phased. We didn't do that when the project started. We did it, say, T-minus one week, because we figured, if it leaks, fine. People know there's something coming out with a new $29.99 cover.”

Pricing of the device itself, though, was a critical piece of strategy. It was kept very close to the vest, making it one of the few product details not pre-populated in the system that morning. Now that it was go time, it wasn’t a matter of simply posting the right price to a single product page. There were also confirmation emails and labels and customer support systems that needed to be updated.

“The worst thing that could happen is somebody sees a bogus price or incorrect price. So until that price is done, that page is not really working. You're basically waiting to know that price is available in all systems that would need price. This propagation delay, it's in the order of minutes. But we were all waiting on that, so that's a big deal.”

Lesson #10: Prep Second-Degree Stakeholders.

While the launch team proper was executing a sequence of tasks they could practically recite in their sleep, a much broader team was on standby, only vaguely aware that they might be looped into the action. “If your service or your app or your product is launching in today's world, there's a bunch of distributed systems that have to play nice,” says Bashir.

While you’ll want to limit top-secret launch information to a small need-to-know group, you do need to give secondary teams a heads-up that something might be coming their way. And fast. “We would prep them and say, ‘Something is happening in the next 72 hours. I need to know who your on-call is, and the best way to get a hold of them. These are the types of things I may ask you to do,’" says Bashir.

Halfway into this particular launch, it was the Customer Reviews team that got the call. Because Kindle with Special Offers wasn’t technically a new product, but a variant of the existing device, it was available at the same Amazon.com/Kindle page. “People were abusing the situation, and started writing reviews of the ads version that hadn’t shipped yet. They’re writing, ‘The ads product is terrible, yada, yada, yada.’ You can’t stop them, because in the system they’re one product.”

Bashir got a hold of his colleagues in Customer Reviews to take down the invalid reviews. But that group came back with their firmly held principle that Amazon is community moderated, and invalid reviews would be down-voted and weeded out over time. “We had to explain that we didn’t have ‘over time,’” says Bashir. “We argued, ‘This person is literally saying the ads-based product is bogus, and that product has not shipped. This review is not legitimate.’”

The Reviews team countered with an alternate solution. “They said, ‘Well, then you should have a separate page that is on pre-order status, where customers can't write reviews.’ And we came back with: ‘We wanted to give people the choice of ads or no ads as a product variant. It's not a separate product.’"

Meanwhile, the clock is ticking. “I'm looking at the time thinking, ‘We’ve got X task coming up next.’ Finally, I told Reviews, ‘You guys have to go solve this.’”

Ultimately, the issue was escalated, the Reviews team was overruled, and the phony customer reviews were removed. In the chaos of a major launch, this subplot highlighted the importance of lesson #11: adopt a culture of disagree and commit.

That’s a core leadership principle at Amazon, but a good philosophy to consider at any company. Anyone can express their viewpoint. But once a decision is made about who is best speaking for the customer in that moment, every other player needs to fall in line.

"Disagree and commit” is shorthand to remind us: it’s not about your team’s interest or your ego. It's about what's the right thing for the customer.

“The question, in this case, was, ‘Are we talking about the reviews experience, which the Reviews team owns, or are we talking about the Kindle sales experience?’ And the decision was that, in that moment, we were talking about the Kindle sales experience. We were just talking about this one instance. Then we agreed to come back to the table and reaffirm that we can't be asking the Reviews team to flag stuff.”

Lesson #12: Be Ready, and Willing, to Hack Your Rules

That raises another important takeaway from Bashir’s experience sending Kindle with Special Offers into the world: launches, particularly the accelerated variety, may require that you bend your own rules.

When it came to the Kindle launch, this played out a number of ways—perhaps most notably with search. When press buzz suddenly sends huge numbers of people looking for your brand-new product, you want to make it as easy as possible for them to find it. Not surprisingly, the launch team wanted any searches for “Kindle” on Amazon.com to return the new product in the top position.

The company’s Search team, though, had a different perspective. “They have their own philosophies and believe that search should be organic. Results that are popular algorithmic-ally will bubble to the top, but they don't want any juicing of search results. If you're the launch team, you're like, ‘I don't care about your philosophy. I want people to see the new product at the top, for at least the next couple of hours.’”

Ultimately, the Search team begrudgingly agreed to manually adjust any wonky search results. “But this is a discussion you have beforehand so you're not worried about it,” says Bashir. That is, to the extent possible, follow lesson #13: pre-decide as much as you can before launch.

There's certain things that have to be decided in the moment because A) you haven't planned for them or they're unforeseen, or B) the data is different on the fly,” says Bashir. “But unless you're going to have some new need or data in the heat of the moment, you should make decisions when it's not a hot moment.”

There was no reason to bring the Search team into the war room. Instead, Bashir and launch leadership hashed out this philosophical difference ahead of time. And when they pre-decided how to handle it, they did so down to the logistical details. “We said, ‘In the event of odd search results, I'm going to page you. If you get this page, this is what you do." Then there was somebody on the Search team who would resolve the issue.

Lesson #14: Don’t Play the Blame Game.

Of course, launches and other major initiatives will almost inevitably surface issues you couldn’t have predicted, which no amount of pre-deciding could have solved. When they do, don’t waste time or energy pointing fingers.

Around the 30-minute mark, Bashir’s Kindle launch hit a snag no one had seen coming. Both the Kindle team and the Amazon Prime team had hacked the site’s main product detail page to add a navigation bar at the top. For users who had both Kindle and Prime accounts, though, those bars were now warring with each other. The Prime team was contacted, and agreed to give theirs up until a code fix could be deployed.

Blame is unproductive, but learning from hiccups is invaluable. “We kept a list of things we could learn from—the ‘How did this happen?’ list—so we added this to it,” says Bashir. That’s lesson #15: track your learnings. In the moment, the ticking clock demands that all non-essential issues be tabled. Logging those issues, though, like all war-room jobs, should be specifically assigned to a single person.

During the Kindle launch, the task fell to Bashir’s boss. “In the heat of the moment, I was registering certain things, not registering other things. Later, we would sit down and talk about the list. Then we held a group retrospective.”

Prioritizing learning is one way to minimize the temptation to point fingers; clearly communicating who’s responsible for what is another. Before launch, every night for two weeks, Bashir sent an email to the entire project team detailing who needed to be in the room for launch, what the reporting looked like, and why each person was there. “There was never any confusion about that. So while people may have been surprised by things, people may have grumbled or voiced their opinions, there was no blame.”

Lesson #16: Always Clean Up After Yourself.

In the end, the launch of Kindle with Special Offers beat expectations, going live in just 37 minutes. That was thanks in no small part to a policy of tabling non-essential issues that couldn’t be resolved quickly—issues that hadn’t gone anywhere once the device was live. “You don’t go home at minute 38,” says Bashir.

Yes, there was a moment to take a breath. Bashir took off his headset, and the team took a moment to appreciate what they’d accomplished. “As soon as it was done, I believe there were donuts or cupcakes,” he says.

Then, the PR crowd left the room to monitor various execs’ interviews. The sales team started checking up on sales volume. And the rest of the team set about cleaning up the messes that had been tabled for later. There was the Kindle page that didn’t play nice with the similarly modified Prime page design, of course. The mobile app didn’t look quite right, and some order confirmations were printing incorrectly.

“We had to finish everything you would do in a normal launch,” says Bashir. “All those things that weren’t your primary concern while the clock was counting down? You still have to fix them.” That brings you to the end of day one. But you’re not really done until every issue that arises out of launch has been resolved.

Bringing It All Together

Before you launch, build in a rapid or partial release option should you need it. Pre-decide everything you can — especially those who will be in the room on launch day. Populate the war room thoughtfully and sparingly; everyone involved should have clear roles and responsibilities. (Senior leaders can be reached, even if they aren’t present.) Leading up to launch, do real-time, full dry runs with the team. When a leak happens, don’t fight it. The launch should be segmented into phases with clear entry and exit criteria — but there should be a series of switches as new scenarios develop. If you’re running the war room, get equipment (headset, standing stool) to be easily heard and seen. Foster a culture of disagree and commit. Track your lessons and clean up after yourself — resolve the issues that had to wait.

“A hardware team declares victory when they have production-ready units. A software team declares victory when they have final bits that go to the factory. A launch team actually doesn't declare victory until after customers have devices in their hands,” says Bashir. “We sit with the customer service team and figure out every single issue that's coming up, and how to resolve it. Then we move on. There’s always another launch to prepare for coming round the bend.”