#compute

Actions Panel

#compute

By Twitter Engineering

Date and time

Thursday, March 31, 2016 · 5:30 - 9pm PDT

Location

@TwitterHQ

1 10th Street San Francisco, CA 94103

Description

The key tenets to building and operating a compute platform are:

  1. Reliability & Scalability: Maintain and scale a compute cluster with no downtime
  2. Efficiency: Utilize hardware resources in an efficient and cost-effective way
  3. Developer Agility: Enabling developers to create, deploy and manage their services in a frictionless manner.

Come and learn how we are building and operating a compute platform at scale. Enjoy a drink, great eats and meet folks who are working on building compute platforms.

About Twitter's Compute Platform:

The platform has grown from managing a few hundred containers to hundreds of thousands of containers, across tens of thousands of hosts. Built on Apache Mesos, Apache Aurora and a suite of internal systems that manage the lifecycle of an application service — Over a thousand services that power Twitter and its associated products run on this platform.

Agenda:

Doors open // 5:30 PM

Food, drinks & networking // 5:30 PM - 6:30 PM

Introduction by Chris Pinkham, VP Engineering // 6:30 PM

Tech talks with Q&A // 6:35 PM - 8:30 PM

Registration closes // 6:45 PM

End of event // 9:00 PM

Speakers:

"Server Provisioning and Management at Scale" by Eric Danielson, Tech Lead - Provisioning Engineering

Twitter operates over tens of thousands of physical servers in its datacenters. Come learn about the systems we use to provision, track, and manage a fleet of this size, and the lessons we've learned building them.

"Managing a large scale compute platform" by David Robinson, Site Reliability Engineer - Compute

Twitter has used Aurora and Mesos to scale from only tens of nodes to tens of thousands. During the process of scaling up, the configuration, deployment, and operational procedures for this system have evolved and improved significantly. This talk will describe how a small team manages a large Mesos+Aurora cluster.

"Chargeback for multi-tenant Infrastructure systems" by Vinu Charanya, Software Engineer and Jessica Yuen, Software Engineer - Cloud Infrastructure Management

We built a chargeback system that incentivizes both the service owner and infrastructure owners to improve resource utilization at Twitter scale. In this talk, we will cover how we built a generic system that helps define chargeable resources, collects utilization metrics, resolves owners and generates billing and utilization reports to correct owners for all multi-tenant infrastructure systems.

"Aurora Workflows" by David Mclaughin, Software Engineer - Compute

Workflows is a new entity in Aurora which represents a set of release targets involved in deploying an artifact to production. The goal is to reduce cognitive overhead from shepherding a deploy from development to production, as well as becoming a key building block for continuous deployment automation in the future. Come hear about how we’re building this and how it’s used within Twitter.

"Twitter Heron on Aurora" by Maosong Fu, Tech Lead - Real-Time Compute

Heron is the next generation real time streaming analytics platform used heavily in Twitter. Heron runs diverse real time analytics applications ranging from counting to real time machine learning. Heron leverages Aurora extensively to run its topologies. In this talk, we will outline how Heron uses Aurora to distribute computation across nodes, discuss the benefits of using Aurora and suggestions on how Aurora can be improved further to support real time applications.

"AWS + Mesos/Aurora at Twitter" by David Hagar, Engineering Manager - Compute (TCDC)

In 2014 TellApart pounced on Aurora as the solution to several problems we were facing in our infrastructure. Hear about our experiences, problems encountered and solved, as well as the next steps: how we are working to make Aurora within AWS an official platform for Twitter, and the challenges of setting up a system that spans public and private clouds.

"Lightning Talk" by Guest Speaker

Organized by

Sales Ended