IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
•
6 likes•1,609 views
Finance market prediction has always been one of the hottest topics in Data Science and Machine Learning. However, the prediction algorithm is just a small piece of the puzzle. Building a data stream pipeline that is constantly combining the latest price info with high volume historical data is extremely challenging using traditional platforms, requiring a lot of code and thinking about how to scale or move into the cloud. This session is going to walk-through the architecture and implementation details of an application built on top of open-source tools that demonstrate how to easily build a stock prediction solution with no source code - except a few lines of R and the UI interface that will consume data through a RESTful endpoint, in real-time. The solution leverages in-memory data grid technology for high-speed ingestion, combining streaming of real-time data and distributed processing for stock indicator algorithms.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
Similar to IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD (20)
More from In-Memory Computing Summit
More from In-Memory Computing Summit (20)
Recently uploaded
Recently uploaded (20)
IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
- 1. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved. A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br 1 William Markito wmarkito@pivotal.io @william_markito
- 3. © Copyright 2014 Pivotal. All rights reserved. It's all about DATA Data Sources Look for patterns Prediction
- 4. © Copyright 2014 Pivotal. All rights reserved.
- 5. © Copyright 2014 Pivotal. All rights reserved. 5 Machine Learning is the answer Neural Networks Clustering Genetic Algorithms
- 6. © Copyright 2014 Pivotal. All rights reserved. Train with historical dataset Apply model to the new input Applying Machine Learning
- 7. Hard to add new data sources Why? Hard to scale Why so hard? Hard to make it real-time
- 8. Traditional models are reactive and static HDFS Data Lake Store Analytics Hard to change Labor intensive Inefficient No real-time information ETL based Data-source specific
- 9. Stream-based, real-time closed-loop analytics are needed HDFSData Lake Expert System / Machine Learning In-Memory Real- Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything
- 10. Info Analysis Look at past trends (for similar input) Evaluate current input Score / Predict Neural Network How can it be addressed?
- 11. Info Analysis Filter [ json ] Neural Network How can it be addressed?
- 12. Info Analysis Filter Enrich Neural Network How can it be addressed?
- 13. Info Analysis Neural Network Filter Enrich Transform How can it be addressed?
- 14. Info Analysis Filter Enrich Transform Neural Network How can it be addressed?
- 15. Info Analysis Filter Enrich Transform Transform Neural Network How can it be addressed?
- 16. Neural Network In-Memory Data Grid Real-time scoring How can it be addressed? Train
- 17. Neural Network In-Memory Data Grid Front-end Update Push How can it be addressed?
- 18. Ingest Transform Sink SpringXD Store / Analyze Fast Data Distributed Computing Predict / Machine Learning Other Sources and Destinations JMS Streaming real-time analytics architecture
- 19. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 20. SpringXD INGEST / SINK PROCESS ANALYZE • Little or no coding required • Dozens of built-in connectors • Seamless integration with Kafka, Sqoop • Create new connectors easily using Spring • Call Spark, Reactor or RxJava • Built-in configurable filtering, splitting and transformation • Out-of-box configurable jobs for batch processing • Import and invoke PMML jobs easily • Call Python, R, Madlib and other tools • Built-in configurable counters and gauges Data Stream Pipelining
- 21. SpringXD XD NodesXD NodesXD NodesXD Nodes Ingest SpringXD Split Filter Transform Sink XD admin XD Nodes Ingest Split Filter Transform Sink Stream Deployment Messaging Scale-Out and HA Architecture
- 22. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 24. Partitioned Regions GemFire Server1 GemFire Server2 Primary'to'redundant'replica1on' Primary 0 2 4 6 Redundant 1 3 5 7 Region A Region B Region A Region B Primary 1 3 5 7 Redundant 0 2 4 6
- 25. Event handling GemFire'Server' Region'A' ' '''''''''' subscrip4on Region'A' pool6name=ServerPool' ' ''''''''' X GemFire'Client'1' pool'"ServerPool"' (with'or'without'' subscrip4ons'enabled) Region'A' pool6name=ServerPool' ' ''''''''' X GemFire'Client'2' pool'"ServerPool"' (with'subscrip4ons'enabled,' 'interest'register'in'X,'receiveValues=true) X Distributed'System Update'/'Create 1 2 3 3 4 X The pool propagates the event to the cache server, where the region is updated. The server distributes the event to its peers and also places it into the subscription queue for Client 2.
- 26. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 27. Neural Networks
- 28. Neural Networks
- 29. medium avg (x+1) relative strength (x) medium avg (x) price(x) Neural Network
- 30. Neural Network
- 31. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 33. Demo Time
- 37. A NEW PLATFORM FOR A NEW ERA