IMCSummit 2015 - Day 1 IT Business Track - From Spark to Ignition
•
3 likes•705 views
Real-time is the next phase of big data. For the modern enterprise, processing and analyzing large volumes of data quickly is integral to success. MemSQL and Spark share design philosophies like memory-optimized data processing and scale out on commodity hardware that enable enterprises to build a real-time time data pipeline, where operational data is always online. This session shares hands-on 'how-to' recipes to build real-time pipelines around Apache Spark, with detailed production examples covered.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to IMCSummit 2015 - Day 1 IT Business Track - From Spark to Ignition
Similar to IMCSummit 2015 - Day 1 IT Business Track - From Spark to Ignition (20)
More from In-Memory Computing Summit
More from In-Memory Computing Summit (20)
Recently uploaded
Recently uploaded (20)
IMCSummit 2015 - Day 1 IT Business Track - From Spark to Ignition
- 1. Fueling Your Business on Real-Time Analytics Eric Frenkiel, MemSQL CEO June 29, 2015 • San Francisco, CA From Spark to Ignition:
- 2. What’s in Store For This Presentation? 1. MemSQL: A real-time database for transactions and analytics 2. Spark Use Cases 3. Example: Geospatial Enhancements
- 3. The real-time database for transactions and analytics MemSQL Story
- 4. MemSQL at a Glance ! Experienced leadership from Facebook, SQL Server, Oracle, Fusion-io ! In-Memory, distributed, relational database ! Solving the Enterprise Architecture Gap ! Horizontal scale-out with modern database innovation ! $50 million in funding
- 5. Four Ways Your DBMS is Holding You Back ! ETL (Extract, Transform, Load) ! Analytic Latency ! Synchronization ! Copies of data Source: Gartner Hybrid/Transactional/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation
- 6. 6 The Real-Time Database for Transactions and Analytics In-Memory Distributed Relational Data CenterSoftware Cloud
- 7. The Real-Time Database for Transactions and Analytics Highest Value Hot Data High Value Cold Data Analytics Transactions Data Loading and Queries Aggregator Nodes Availability Group 1 Availability Group 2 Cluster
- 8. Gartner Identifies Emerging Category: HTAP (Hybrid Transactional/Analytical Processing) “HTAP will enable business leaders to perform…much more advanced and sophisticated real-time analysis of their business data than with traditional architectures.” Download at: memsql.com/gartner
- 9. Simple ! Standard SQL ! Transactions and analytics in one database ! Behind the firewall or on the cloud ! Flexible integrations (Hadoop, Spark, SQL)
- 10. Fast ! Extremely low-latency queries ! Massive parallel transaction capacity ! Lock-free, shared-nothing architecture
- 11. Scalable ! Scales out on cloud and commodity hardware ! Deploys to thousands of machines ! True linear scaling
- 12. MemSQL Product Ecosystem In-Memory Applications Transactions and Analytics Dashboards HadoopAmazon S3 ODBC, JDBC, .NET Connectors MemSQL Loader Advanced Analytics Wire-protocol compatibility Databases and Data Warehouses Streaming
- 13. Spark Use Cases
- 14. Spark Data Processing Framework Intuitive, concise, and expressive operations needed for analytics Spark SQL Spark Streaming Mllib (machine learning) GraphX (graph) Apache Spark
- 15. Cluster-wide Parallelization | Bi-Directional Understanding MemSQL and Spark
- 16. Spark with MemSQL MemSQL Spark Connector enables the real-time trinity Message Queue Transformation Data Serving Programming libraries Persistence Application platform End-to-End Data Pipeline Under One Second
- 17. MemSQL and Spark Use Cases ! Operationalize models built in Spark ! Stream and event processing ! Live dashboards and automated reports ! Extend MemSQL analytics
- 18. Operationalize Models Built in Spark ! Process in Spark, persist to MemSQL ! Go to production and iterate faster Enterprise Consumption Data into Spark Model Creation Model Persistence
- 19. Stream and Event Processing ! Structure event data on the fly ! Pass to MemSQL for persistent, queryable format Enterprise Consumption Real-time Streaming Data Data Transformation Persistent, Queryable Format
- 20. Real-Time Analytics at Pinterest ! Higher performance event logging ! Reliable log transport and storage ! Faster query execution on real-time data Enterprise Consumption Message Queue KafkaApp Singer Secor Spark, MemSQL, Application Transform RT Analytics, Data Serve 50,000 pins/sec
- 22. Live Dashboards and Automated Reports ! Serve live dashboards from MemSQL ! Run custom reports on live data with Spark Live Dashboards Custom Reporting Access to Live Production Data SQL Transactions and Analytics
- 23. Extend MemSQL Analytics ! The freshest data for analysis in Spark ! Load from MemSQL to Spark and write results on return Access to Live Production Data Real-time Replica Applications, Data Streams Interactive Analytics, Machine Learning
- 24. MemCity ! Capturing energy consumption data from 1.4 million households ! 8 devices per household ! 186,000 events per minute ! AWS hardware costs at $2.35 per hour
- 27. Geospatial Challenge ! Commercial applications now geo-enabled ! Location is everywhere ! Lots of insight possible ! Traditionally geo is processed separately ! Real need for integrated geospatial at scale
- 28. MemSQL Geospatial ! Points, Lines, and Polygons ! Topological filters ! Measurement functions
- 29. MemSQL Geospatial ! BILLIONS of objects ! Sub-second latency ! Geo data is first-class citizen ! Geo + Simplicity + Speed + Scale
- 30. Real-Time Geospatial Location Intelligence ! Sample from 170 million taxi trips ! Real-time ingest ! Concurrent queries in fractions of a second ! Unlimited number of geographic views ! Simple queries while simultaneously ingesting data
- 32. A database so scalable that everyone can use it. UNLIMITED scale and capacity Free FOREVER MemSQL 4 Community Edition
- 33. Thank You! Visit the MemSQL Booth #4 MemCity Showcase GiveawaysGames *