Modeling a game with melody recognition and machine training using ClojureScript and real instruments

Being great at a first-person shooter game requires a lot of hand-eye coordination and skill with the mouse. Being great at an MMORPG requires a working knowledge of a vast variety of keyboard shortcuts. Being great at a fighting game requires speed and skill with the gamepad.

The underlying philosophy behind this experiment is to showcase that you can potentially take the input to any stringed instrument (with a pickup) lying around in your house, to use to design a game that requires as much skill (if not more) as playing any other game with any other kind of input device. By treating the instrument as a game controller, and moving the onus of deriving quality from the playing into the system, we can harness some of the basic qualities – speed and melody – of the instrument to make our game more interesting and a lot more unique and fun.

The idea is to go as far away from being a Guitar Hero derivative as possible, and closer to a system that treats it as just another input device. The game should not be about playing guitars or any kind of music, but should be about playing any other regular game. Unlike other Musical Instrument Games™, we want ours to work for anyone, anywhere. So we're going to use a simple semi-electric guitar plugged into the microphone line-in and demonstrate this on the browser using the Web Audio Interface.

We'll go through a few papers and look at implementing some of the algorithms to pitch / note detect in near real-time, process the input to make sense of it, run it through a few simple constraints and algorithms to recognise what a melody might be like and eventually explore how we can make the system intelligent enough to train itself based on the player and the styles so that we can generate smarter and harder levels for our game.


Outline/Structure of the Demonstration

Before we deep dive into the implementation, we're going to discuss music in general. The game is rather a secondary goal for this exercise and the outcome will serve merely as a demonstration of possibility.

Music Theory

We'll cover the basic building blocks of music from ground-up. Starting from,

  • Note
  • Tuning
  • Intervals
  • Temperaments
  • Scales
  • Chords

and how these apply to various traditions of music (western, contemporary, carnatic etc.)


We'll examine what fundamental frequencies, harmonics and the frequency-ratio characteristics (twelfth-root of 2 etc.) of these different styles are like.


We'll go through these papers and discuss some of the pros and cons of the each of these algorithms. We'll explore how the accuracy of each of them vary based on the kind of signals we receive. Polyphonic sounds for any proper guitar playing holds a lot of frequency fluctuations that are hard to deal with, so the robustness for pitch detection is still more or less an unsolved problem.

ClojureScript Wrapper

I've been working on a fork of the ClojureScript library called Hum. The original is still small and lacks a few APIs that we require, for e.g. AnalyserNode


Even the high-end hardware at this point fails to properly do chords. As you can design chord notes to hit directly on the harmonics of the fundamental frequency. We'll talk about some simple solutions that can potentially make our output better, for e.g.: putting six piezos on each string of the guitar that can send MIDI data.


We'll devise a simple Fleuch-esque game where a space rover can shoot in a 360° arc around itself to destroy incoming droids. The speed and attack radius of this rover is controlled by how melodic the player plays. Here's a simple shot of what this would look like:


We can back this up with a persistent connection to a server that takes all this frequency data to bucket our notes into trying to determine what a good melody might be. This is obviously a very non-deterministic problem, and an objectively correct solution is impossible.

We could start by bucketing our melody engine into good, reasonable and bad templates. Each of these buckets would have a set of constraints that we would run our notes through.

  • Pre-feed frequency data from popular music for comparisons.
  • Playing a series of notes in a certain window frame that usually sound good (based on some simple rules of musical scale)
  • Not let the player beat the game by merely playing the same nice-sounding notes / scales over and over again

Learning Outcome

ClojureScript, Music Theory, Web Audio API

Target Audience

Clojure / ClojureScript enthusiasts, music tweaking, DIY games

schedule Submitted 6 years ago

Public Feedback

    • Morten Kromberg

      Morten Kromberg / Jay Foad - Array Oriented Functional Programming with Dyalog

      480 Mins

      Dyalog is a modern, array-first, multi-paradigm programming language, which supports functional, object-oriented and imperative programming based on an APL language kernel. During the morning and early afternoon, Dyalog's functional language guru John Scholes will introduce the fundamentals of functional programming in APL. At the end of the day, Dyalog CTO Morten Kromberg will round off with an application development session, showing how a simple web application with a HTML5/JS front-end and a RESTful web service can be used to deliver the power of APL to end users, or as embeddable components for other application tools. The "hands on" sections of the workshop can be followed under Mac OS X, Linux, or Microsoft Windows.

    • Abhinav Sarkar

      Abhinav Sarkar - Introduction to Concurrency in Haskell

      Abhinav Sarkar
      Abhinav Sarkar
      Software Engineer
      schedule 6 years ago
      Sold Out!
      45 Mins

      Haskell is widely known as a functional programming language which is great to write pure (non I/O) code and immutable data structures. But it is not well known that it has a great support for a wide variety of concurrency models too.

      In this talk we shall explore the various concurrency models and primitives supported by Haskell including green threads, mutable variables, channels and software transactional memory. We shall see how the purity, laziness, and the strong type system of Haskell come together, and help write clean and modular concurrent code. We shall explore these by writing a mutli-user chat server with support for private and channel chatting.


    • Tanmai Gopal
      Tanmai Gopal
      Engineering Head, Founder
      34 Cross
      schedule 6 years ago
      Sold Out!
      45 Mins
      Experience Report

      From the perspective of one of the few companies in the country using Haskell in production (, we will talk about the journey of using Haskell as a programming language to build and maintain real world software.
      From the burden of choosing a language that probably has a single digit number of active developers in the country, to
      the joys of a small team churning out code that just keeps on working.

      We'll go over how we created a team, organized work ("software development practices"), architecting code, tooling up, testing, performance tweaking and finally deploying a Haskell codebase. We hope that Haskell developers with some prior experience will be able to take away
      something new about the Haskell ecosystem, and those interested in just functional programming will see tangible value in how certain features unique to Haskell have an impact.

      Expect a case study, a review of some cutting-edge tools and how to use them, and some Haskell code samples that will make you grunt in geeky approval.

    • Ravi Mohan

      Ravi Mohan - Building a General Game Playing Engine with OCaml and Erlang

      Ravi Mohan
      Ravi Mohan
      schedule 6 years ago
      Sold Out!
      90 Mins
      Experience Report

      "General Game Playing  is the design of artificial intelligence programs that play more than one game"  Wikipedia [1]

      In other words, one program has to be able to play (and win!) multiple games ( Chess, checkers, Go, Othello etc).

      Summary: This experience report is about my  "Forever Project" [2,3,4] to build such a system in OCaml, and the problems (theoretical and practical) and will include a demo of the program, warts and all.

      Detail: GGP is an active CS research area [5]Annual competitions are held every year where  ggp programs compete against each other[6], playing games which they have never seen before (the rules are supplied as an input file for the system to consume five minutes before the match begins).

      Over the last year, in my non-existent spare time, I've been building a GGP engine to compete in a GGP competition.

      Most such engines are written in Java, C++  etc.. I'm using OCaml for the game playing agent, and Manoj is building the backend in Erlang.

      As mentioned above this is our "Forever Project"  We've made some decent progress.

      Come, see, laugh, jeer!

    • Nivedita Priyadarshini

      Nivedita Priyadarshini / Srihari Sriraman - Building an Experimentation Platform in Clojure

      45 Mins
      Experience Report

      Here's a markdown version of this proposal:

      Over the last year and half at Staples SparX, we built a multivariate testing platform as a service. It satisfies an SLA of 10ms at 99.9th percentile, services all of Staples' experimentation from a single machine, is simulation tested, and is written in Clojure.

      We'll give an introduction to the Experimentation domain, design of experiments and our battle in attaining statistical significance with constrained traffic. We will share our experiences in loading and reporting over months of data in Datomic, using Clojure to grow a resilient postgres cluster, using a homegrown jdbc driver, interesting anecdotes, and OLAP solutions with ETL built in Clojure using core.async. Expect to see references to google white papers, latency and network graphs, histograms, comparison tables and an eyeful of clojure code.

    • Srihari Sriraman

      Srihari Sriraman - Carnatic music synthesis in Clojure

      Srihari Sriraman
      Srihari Sriraman
      Partner, Engineer
      schedule 6 years ago
      Sold Out!
      45 Mins

      Here's a markdown version of this proposal:

      I'm building a carnatic music (south indian classical music) synthesizer using Clojure and Overtone. Here's the library I'm building.

      Carnatic music is different fundamentally in the movements of pitch ('Gamakams'), and needs to be modelled differently than most MIDI music out there today. The library now, can understand prescriptive carnatic music notation, build finer abstractions from that, and play back a not-so-terrible synth version with a plethora of configurable inputs.

      The concept of 'Gamakam' (the transitions between notes) is central, so I'll go into it's guts and show how the library piggybacks on recent work on PASR (pitch-attack-sustain-release) and Dual-PASR ('stage' and 'dance' PASR components) transcriptions. I'll also put forth a path I see for fully synthetic carnatic music based on machine learning 'Ragams' (melodic modes).

      I'll give a detailed demo of the synthetic music on the repl through the talk, and do a live voice vs synth comparison in the end.