High Performance Data Mart using Event Plots

location_city Bengaluru schedule Nov 17th 05:00 - 05:20 PM place Regal people 3 Interested

Reporting data for finite state machine transactions like calls require exhaustive data mining into 10s and 100s of parameters for 1000s of calls per hour. These postmortems require heavy processing and can demand millions of maps per reduction. The presentation is a case study and demonstration of near-realtime asynchronous event processing based plotting of events and states of FSMs. Thus creating semi-cooked event plots delivering lightening fast reports with extensive customizability.


Outline/Structure of the Demonstration

The session would start with the problem definition of the case study, followed by the solution design and a demonstration of the technique at the end the session.

  1. Presentation (10 min ~ 15 min)
    1. Domain overview - Cloud-based Call Center Solution
    2. Problem Definition - Reports using traditional Datamart approach - short falls - challenges
    3. Overview of Data Plots - Erlang Messaging - Event Maps - Mnesia/ETS (memory & disc) - Mnesia/eleveldb(disc only)
    4. Reporting - Scanning Events - Data Aggregation - publish - stream - MapReduce
  2. Demonstration (10 min ~ 15 min)
    1. Pre-populated call center event plots (mnesia, 200 GB offline / 1TB online)
    2. Access the web based report dashboard and demonstrate performance of existing reports
    3. Modify any existing report to demonstrate customization
    4. Simulate a bunch of call events and demonstrate the report streaming
  3. Questions & Discussions (5 min ~ offline)

Learning Outcome

The participant would be able to appreciate the technique, the functional design aspect and the Erlang/OTP efficiency.

The participant can also get the exposure to the data mart and reporting solutions.

Target Audience

Functional Programmers, Architects and Data Mart Engineers

Prerequisites for Attendees

The participants should be aware of async messaging, map-reduce algorithm and basics of functional programming.


schedule Submitted 4 years ago

Public Feedback

    • Eric Torreborre

      Eric Torreborre - Practical eff monad for micro-services

      Eric Torreborre
      Eric Torreborre
      Sr. Software Engineer
      schedule 4 years ago
      Sold Out!
      45 Mins

      This talk presents how we use the Eff monad at Zalando to structure a REST microservice. Even when programming with functions, modules and components still matter, and I will present a new approach for dependency injection using the Reader monad (classic) and tree rewriting (what?). Then we will see how to specify more precisely several components (authentication, request to other services and database access) with the Eff monad and support additional concerns like logging or asynchronous execution.

      Once you get the basic idea, the possibilities are endless! Rate limiting is just another component and another set of effects involving time and state, testing becomes really clean and easy.

    • Aaron W Hsu

      Aaron W Hsu - Design Patterns vs. Anti-pattern in APL

      45 Mins

      APL is a notorious language with a reputation for being "write only" and difficult to learn. Yet, many people consider APL to be their single most productive, advantageous tool that they hope never to give up. In particular, it is generally observed that many computer scientists have a poor opinion of the usability of APL, while many data scientists and domain experts find the language to be exceptionally usable.

      This session focuses on the experienced programmer's struggle to gain real facility with APL beyond the basic, trivial understanding of its semantics and syntax (which can be taught in a few hours). We will observe this struggle through a "human-centered" experience analysis identifying a set of Pattern/Anti-pattern tensions that are at the heart of the intermediate APL learning "wall." By examining the experience of thinking and working with APL code versus regular code, we can identify principles of APL coding practice that directly oppose the widely taught and embraced best practices of the broader programming community. We'll see not only what these principles are, but also why they survive in APL and how they contribute to the positive development cycle of an experienced APL programmer, instead of the negative impact such practices usually have when writing in other languages.

      Understanding these practices will provide a focus point for the discussion of programming experience design and the use of concise array language, as well as provide a structure for becoming better able to write, read, and think in APL expert.

    • 45 Mins

      In Indian classical music, we have Jugalbandi, where two lead musicians or vocalist engage in a playful competition. There is jugalbandi between Flutist and a Percussionist (say using Tabla as the instrument). Compositions rendered by flutist will be heard by the percussionist and will replay the same notes, but now on Tabla and vice-versa is also possible.

      In a similar way, we will perform Code Jugalbandi to see how the solution looks using different programming languages and paradigms.This time the focus of code jugalbandi will be on solutioning in different paradigms - object-oriented or functional programming and array-oriented paradigms.

      During the session, Morten and Dhaval will take turns at coding the same problem using different languages and paradigms. There would be multiple such rounds during the Jugalbandi.

    • Tony Morris

      Tony Morris - Functional Programming in Aviation

      Tony Morris
      Tony Morris
      Software Engineer
      Simple Machines
      schedule 4 years ago
      Sold Out!
      45 Mins
      Case Study

      In this talk & demo, we have a look at some of the low-hanging problems in general aviation and how functional programming can be applied to provide significant improvements in efficiency and air safety. The current solutions to problems such as navigation, traffic/terrain collision avoidance and weight/balance calculations will be demonstrated to the audience, mostly for amusement. More seriously, we will have a look at the legacy that has led to the way things are, and how to improve by applying our programming skills.

      We will look at:

      • how aviation safety is regulated.
      • how aeronautical services are provided to flight operators.
      • how aeronautical navigation is conducted and regulated.
      • how the weight and balance for a flight is conducted.
      • the methods by which aircraft and ground coordinate between each other.

      We will see:

      • some real (and basic) data management problems in aviation, that very obviously threaten safety, then solve them, using programming.
      • we will see a live demonstration of aeronautical navigation methods, investigate incident reports where lives were lost as a result, and consider how our programming skills can yield improvements, possibly even save lives.
      • we will conduct a real weight&balance calculation for a flight, then once hilarity inevitably ensues, we will look at the problems that arise by this method, then solve them using data structures and functional programming. Some
        of these practical problems are obvious, even to a non-aviator, and the predictable incident reports are the end result.
      • finally, we will have a look at a live demonstration of a software defined radio (SDR), receiving ADS-B transmissions from aircraft (live), an AHRS implementation and GNSS receiver using off-the-shelf, low-cost parts. We will look at why these instruments are helpful to aircraft pilots and interact with that device using the Haskell programming language.
    • Aloïs Cochard

      Aloïs Cochard / Tony Morris - Introduction to Functional Programming using Haskell

      480 Mins

      We will be spending the day learning the fundamentals of Functional Programming (FP) using the Haskell programming language. The exercise material will be a condensed selection of the NICTA/course which is regularly held in Australia over three days.

      This one day session is targeted to experienced industry programmers who are looking to break into Functional Programming and develop the rudimentary skills and techniques that enable continued independent study. A refresher on Haskell syntax will be provided, however, it is highly recommended to practice with the syntax and development tools prior to obtain the best outcome for the day.

      You will be required to bring a suitable development machine (portable) for working through the exercises. You will also need to install Glasgow Haskell Compiler (http://www.haskell.org/ghc/) version 7.8 or higher on that machine prior to the day.

    • Shri Vignesh Balasubramaniam

      Shri Vignesh Balasubramaniam - Understanding some basic FP constructs by writing a simple Programming Language

      Shri Vignesh Balasubramaniam
      Shri Vignesh Balasubramaniam
      Lead Engineer
      schedule 4 years ago
      Sold Out!
      45 Mins

      What I cannot create, I do not understand - Richard Feynman

      Most of the time, we end up being consumers of a language. Language creation is not something which many of us do as part of our daily job. And as such, we become consumers of using various concepts like types of a language, functions-as-first-class citizens, closures, let expressions, etc. But what are these? What is semantics behind them? And how do language creators enable a language to have these? Once we understand what these mean, we go beyond the language specific implementations and can be true polyglots who can think in terms of these composable abstractions.

      This beginner focussed talk+demonstration hopes to clarify few fundamental concepts by building a language from the ground up. This talk has its foundations in a couple of academic courses which I got to be part of during 2012/2013.

      Using Racket(a variant of Scheme) as a vehicle, a hypothesized programming language (MathL) is introduced with its specifications, syntax, and rules. And slowly this language is built up with more features.The language is realized via an interpreter/executor. [We would not be discussing compilers here].
      Some tests also would be done which would help to verify the language implementation.




    • Manoj Govindan

      Manoj Govindan - Learning Computer Architecture

      Manoj Govindan
      Manoj Govindan
      Director, Engineering
      schedule 4 years ago
      Sold Out!
      45 Mins
      Experience Report

      Computer Architecture begins with electronics. As computers are built layer upon layer starting with primitive gates baked into silicon, followed by chips and logic gates, hardware platform including ALU, RAM, cache etc., and so on, so is the study of computer architecture a study of each of these layers of abstraction.

      Students of computer architecture usually rely on simulators, often written in Java or similar imperative/object oriented languages, to aid their learning. Here are some examples of such a program:

      Simulating XOR gate

      As an Erlang/functional programming enthusiast studying computer architecture I decided to write my own equivalent tools in Erlang and TypeScript as I went about my learning. I am studying computer architecture and simultaneously writing the required simulators and other tooling in Erlang and TypeScript.

      Building such emulators and tools in Erlang and Typescript results in distinctly different architectures. I'll be demoing the simulator I built and talking about lessons learned doing this.

    • Abhiroop Sarkar

      Abhiroop Sarkar - Wormholes and teleporting with Transient.

      45 Mins

      The transient library in Haskell/Eta composes arbitrary pieces of software in concurrent, parallel and sequential settings using standard Haskell/Eta combinators like applicative, alternative, bind etc. It provides algebraic and monadically composable primitives for concurrency, parallelism, event handling, transactions and multithreading

      It liberates the programmers from dealing with state machines, objects frameworks, routes, configurations and callbacks. Transient expresses business functionalities as compact category theoretical expressions which compose. These compositions are verified during compilation, although the code might be executing in different machines.

      It presents a fresh new approach to programming concurrent and distributed systems.


    • Ryan Lemmer

      Ryan Lemmer - Haskell Design Patterns

      480 Mins

      This is an experiential workshop, we will code together, and explore ideas from first principles. You will need GHC 8.*

      The workshop is pitched at Haskell enthusiasts (you don't have to be hardcore at Haskell, I am not!). We'll be writing code together and apart, reflecting on code and ideas, and getting into and out of trouble.

      Patterns for IO

      Experience the 3 styles of IO programming in Haskell, in the order in which they evolved:

      * imperative style (the status quo)
      * Lazy IO and its limitations
      * Iteratee IO, the solution to Lazy IO

      As we compare the different styles, we will pay attention to

      * what controls evaluation
      * space and resource management
      * the relationship between producer and consumer

      From Foldable, Traversable to Lenses

      We follow the evolution of fold and map on lists, to their generalisation over arbitrary types with the Foldable and Traversable type-classes.

      The Lens library raises Foldable and Traversable to an even higher level of abstraction. We will see how the Lens library gives a unified vocabulary for traversal of arbitrary data structures.

      Functor, Applicative, Arrow, Monad

      These data types fall on a spectrum of power and generality (Functor being the most general, and Monad the most powerful). By experiencing how these types differ in the way they compose, we can gain a clearer understanding of where and how each type may be used.

      Dependent Haskell / Type-level programming 101 (bonus material)

      In modern Haskell, the line between types and functions have blurred, types have become as powerful and expressive as functions. Since we can write functions on type-level, we can do whatever we like with types! Almost.

      Let's write some type-level code, and see why Dependently Typed Programming matters.

      Datatype Generic Programming (bonus material)

      This style of Haskell programming is based on a simple technique: instead of defining functions for ad-hoc types, we deconstruct our types into a more fundamental type representation, and then write generic functions against the lower-level type. These generic functions are impervious to changes in the original, higher-level data types.

      We will experience datatype generic programming from first principles, and along the way uncover the key ingredient in the magic sauce called GHC.Generics.