This is a lessons learned post concerning a longstanding game idea of mine that I attempted to make, the progress I made doing so, and the reasons why I ultimately put it down to move on to other projects. The code for the game, as far as I got with it, is here: github.com/millerpeterson/sound-maze-game.
The basic idea of the game is that you are trapped in a dark maze, and you are being hunted by an unseen entity. You must find the exit before the thing finds you, but the darkness means you have only auditory clues as to which direction to take. The unique selling point for the game would be its minimal graphics and user interface: you would play the game on the Novation Launchpad MIDI controller, an 8 x 8 grid of LED push buttons. With these stripped down visual possibilities, 3D audio would be the immersive focus of the game.
I had other vague notions of making it a rogue-like with procedurally generated mazes, and evolving AI behaviors, though I never fleshed those ideas out. In the end, I implemented a simple proof of concept where you can walk through a maze and hear a 3D sound source relative to you. If that doesn’t sound too impressive, that’s because it isn’t - I’m writing this to reflect on why it took me so long build this prototype. In short, my two main take-away lessons are:
Don’t make your game engine a distributed system.
Validate your gameplay ideas before you start building a game engine in the first place.
Both of those may be obvious! In my case, however, I had this game concept that I thought would be awesome brewing for over 5 years, and then a few months of experimentation was all it took to see some major flaws in both my technical approach and the game concept itself.
Working with a Distributed Game Engine
Before I came up with the idea for this game, it had long been my conviction that sound design is an under-developed aspect of games. Most technical innovation in games seem to be focused on graphics, with sound being a more marginal consideration. Of course, AAA games like Call of Duty have amazing sound design, but the relationship between the sound and gameplay hasn’t changed much in the past decade. It would be interesting, I thought, to make a game where the focus was on creating an experience primarily with sound, rather than having sound play second fiddle to graphics.
To achieve this sound-first approach to game design, I reckoned it would make sense to use a system that would allow me to rapidly prototype different sound generators. At first I looked at Max/MSP for this, as it lets you quickly experiment with synthesizer designs, and I have some experience integrating it with other programming environments (see my post on connecting Max/MSP to Clojure below). Integrating more traditional programming environments with Max/MSP has gotten a bit easier, as recent releases of Max/MSP include better capabilities for working with JSON data.
Without too much effort I was able to pass structured data from my main game logic process running in Clojure to the audio engine in Max/MSP by sending JSON over UDP. The Clojure process would compute a “view” of the game state, then pass this over the wire to the audio engine, which would make sure the correct sounds were playing.
Having the amazing interactive development experience of Clojure mixed with the ability to rapidly prototype the sounds in Max/MSP, I was pumped to start programming my game! But as it turns out, there was a fair bit of friction between the worlds of the game logic in the JVM and Max/MSP. For example, if I failed to clean up network resources in my Clojure app, I would need to rebind both the game logic and the sound engine to a new UDP port.
In the end, it would typically take me 10 - 15 minutes each time I worked on the project to get the two pieces talking to each other and the state properly synchronized between them. Given that I work on projects like this in my spare time, often on weeknights, that is a significant price to pay in terms of productivity. The much-discussed lengthy startup time for Clojure REPLs, to my chagrin, did start to bother me towards the end of working on this project.
This brings me to my first take-away from this project: don’t make your weird side-project game idea into a distributed system if you don’t absolutely need to! You will unnecessarily waste your precious creative energy fighting with configuration and sync issues that have nothing to do with the idea you are trying to realize.
But even more fundametally: is the idea you are trying to realize any good?
Building the Wrong Thing
I’ve heard it said that often the problem with software is not that it’s technically lacking in some way, but rather that the fundamentally idea behind it is ill-conceived. As much as I hate to admit it, I believe this is true of this project.
I had it in my head that a game played in the dark with 3D sound cues was somehow a new idea that would be immersive in a novel way. In fact, FPS games nowadays already make excellent use of 3D sound to provide an immersive gaming experience. Recently while playing Prey (a survival shooter in the tradition of BioShock) I found myself creeping around a space station, narrowly avoiding roaming aliens using only the sounds of their footsteps and creepy vocalizations. In conjunction with the AAA graphics, this is an incredibly engrossing experience.
The detail and polish of the interactivity matter for things like this. The rough edges on the sound module I was using to position sounds in 3D space really detracted from the experience, and it began to seem pointless to fight with this given that high-quality 3D sound positioning is now available with basically no effort in game engines.
My second take-away from this project is therefore the following: take the quickest path to validating you game / application idea before implementing a complex technical apparatus to support it. Had I started with a rougher prototype using a technically simpler approach, I may have realized quicker that the whole plan wasn’t really that compelling.
In the pasttwo posts I’ve been describing a project to play byte beat formulas
in the browser. I had a way to play them using the Web Audio API, and a large
body of formulas to start out with, built by parsing existing C expressions into
Clojure ASTs.
Byte beat formulas are really fun to mess around with, but most programs give you
a text-editor style interface to modify the formulas. I wanted to create a way to
interact with the formulas that was more accessible, and more amenable to quick
experimentation.
Previously I had come across this blog post,
about genetic programming in Clojure. What inspired me was the idea of “breeding”
two Clojure functions by swapping branches of their respective ASTs. This seemed like
an interesting way to explore the space of possible formulas: take two byte beats that you
like, and see what their offspring would sound like.
If you use zippers,
a functional data structure well-suited to editing trees, the
genetic programming operations are simple to implement. Here is my implementation of
crossover:
(defnreplace-branch"Replace a formula branch rooted at l with a formula branch rooted at r."[lr](zip/root(zip/replacel(zip/noder))))(defncrossover"Return a breed two formula, switching a random branch in l
with a random branch in r."[l-formr-form](let[l(rand-nth(op-tree-locsl-form))r(rand-nth(op-tree-locsr-form))](replace-branchlr)))
op-tree-locs just filters the list of tree nodes to those that represent operators
in the byte beat formulas (and thus valid sites for swapping branches).
I also implemented operations that are akin to genetic mutation:
Mutate: randomly change a constant in the formula.
Complexify: grow a formula tree by replacing a constant with a sub-expression.
Simplify: shrink a formula tree by replacing a sub-expression with a constant.
It’s not exactly genetic programming, unless you consider your own aesthetic
preferences to be a fitness function. That said, I think the app achieves roughly what I set
out to do: it’s a quick and easy way to play around with byte beat formulas in your browser.
As I outlined in my previous post,
I had set out to create a byte beat player for the web
using ClojureScript and the Web Audio API. Ideally this would have some novel ways to generate new
formulas. My main complaint with existing byte beat synth programs is that the interface is
basically a text editor and a C compiler. You should be able to have fun with byte beats without
having to understand what is actually going on in byte beat formulas.
To start, I needed to figure out how to play a byte beat formula in the browser. From past tinkering
with the Web Audio API, I knew that there was an audio graph element called a
ScriptProcessorNode,
which could be used to create samples by evaluating an arbitrary function. The way this works is that
your ScriptProcessorNode will execute a callback which is passed an audio buffer, and the callback
fills up the buffer with sample values. Here is what my ClojureScript implementation looked like;
this function returns the callback that will receive ScriptProcessorNode events:
(defnaudio-event-processor[clock-refsample-genrate-ratio]"A function that fills the audio buffer in an audioprocess event with samples from using
sample-gen ranging over a clock's values, rate adjusted according to a sample rate ratio."(fn[ap-event](let[out-buff(.-outputBufferap-event)buffer-sample-gen(fn[buff-index](let[clock-rel-t(+buff-index(derefclock-ref))rate-adjusted-t(Math/floor(/clock-rel-trate-ratio))](sample-genrate-adjusted-t)))](fill-buffer!out-buff(compfolded-ampbuffer-sample-gen))(swap!clock-ref#(+%(.-lengthout-buff))))))
sample-gen is a function that would take one argument, essentially the “t” in byte beat formulas.
There’s a bit of math using rate-ratio - this allows you to stretch or compress a
formula by manipulating its sample rate. This function mutates a global clock reference to increment
the byte beat “t” value; if I had to do it again I would try to do this with a closure.
Now that I had a way to play the byte beats, I wanted to try out some of the awesome existing byte beat
formulas.
At first I tried manually re-writing some of the famous ones in ClojureScript. Here’s
the ClojureScript version of the famous t * ((t>>12 | t>>8) & 63 & t>>4):
Manually coding these rapidly grew tedious, so I decided I needed a way to parse the C expressions
and convert them into ClojureScript expressions. A bit of searching lead me to
Instaparse, an awesome library for generating parsers
for language grammars. It works in the browser through the instaparse-cljs port.
Arithmetic is a classic instructional example for CFGs, so I was able to
get oriented quickly. Through some trial and error I figured out how to get the operator precedence
right for the byte beat grammar. Here is what I came up with:
expr = bitwise-or
<bitwise-or> = bitwise-xor | bit-or
bit-or = bitwise-or <'|'> bitwise-xor
<bitwise-xor> = bitwise-and | bit-xor
bit-xor = bitwise-xor <'^'> bitwise-and
<bitwise-and> = bit-shift | bit-and
bit-and = bitwise-and <'&'> bit-shift
<bit-shift> = add-sub | shift-left | shift-right
shift-left = bit-shift <'<<'> add-sub
shift-right = bit-shift <'>>'> add-sub
<add-sub> = mult-div | add | sub
add = add-sub <'+'> mult-div
sub = add-sub <'-'> mult-div
<mult-div> = term | mult | div | mod
mult = mult-div <'*'> term
div = mult-div <'/'> term
mod = mult-div <'%'> term
int-cast = <'(int)('> bitwise-or <')'>
sin = <'sin('> bitwise-or <')'>
tan = <'tan('> bitwise-or <')'>
<term> = number | variable | <'('> bitwise-or <')'> | int-cast | sin | tan
variable = 't'
<number> = floating | integer | hex
floating = #'-{0,1}\d+\.\d+'
integer = #'-{0,1}\d+'
hex = #'-{0,1}0x(\d|[a-f]|[A-F])+'
Now I was able to parse and play all the cool byte beat formulas people have come
up with. It’s pretty fantastic that ClojureScript and Instaparse let you do all this
in the browser!
In the next post I’ll get into how I used genetic programming operations with the
byte beat formula syntax trees to generate formulas.
Consider for a moment the following small fragment of C code,
which outputs a sequence of bytes based on an ever-incrementing counter (t):
for(t=0;;t++){putchar(t*((t>>12|t>>8)&63&t>>4));}
Now imagine this is generating samples to be passed to a DAC to create sound
(e.g. pipe the output of this program to /dev/dsp on a Unix system). What
do you think this would sound like? Probably just a bunch of random crunchy
noise, right? The formula, a seemingly arbitrary combination of mostly bitwise
operators, doesn’t really resemble a typical synthesis algorithm. There are no
obviously periodic aspects to the formula; there are no sine or cosine functions.
And yet if you actually try running this, you’ll see that it produces a musical,
chiptune-style melody that exhibits a surprising amount of variety, both in terms
of its melody and timbre, as it evolves over time. This formula and many others
were originally discovered in the demoscene community,
principally by Viznut, and documented in
a series of blog posts
and YouTube videos; the first one is worth checking out if you haven’t seen this already:
This phenomena has come to be known as byte beats, and people have been exploring
the musical possibilities of these short formulas through a variety of applications
(Bitwiz and GlitchMachine for iOS are some of my favourites). You can try some out
on the web here.
Personally, I am fascinated by two aspects of these formulas:
You can pack an enormous amount of sonic variety in an extremely concise formula;
byte beats are generative in the best sense.
The synthesis technique is somewhat mysterious, and mixes concerns about melody and
timbre.
If you are curious about why these formulas sound the way they do, Viznut has a fairly
in-depth analysis of some of them in this blog post.
To simplify, the periodicity that is required for harmonic sounds comes from the implicit modular “wrap-around”
that occurs when you 8-bit downsample the evaluated value of a formula. The melodies
come from the bitshift operators, which are tantamount to multiplication or division by powers
of two, and thus create the kinds of whole-number interval relations between frequencies
that exist in melodies.
I’m not particularly interested in engineering byte beat formulas to
make conventional sounds or melodies, although I can see how this would be a really cool
“compression algorithm” for the soundtrack to a 64kB demo. For me, the fun is in playing
around with these formulas in a trial-and-error kind of way, and just seeing what kind of
sounds you can produce.
For the past few months, I’ve been working on a browser-based program for exploring the
sonic possibilities of byte beats in a fun and intuitive way. The idea is to use genetic
programming to generate novel and interesting byte beat formulas. I’m coding the project
using ClojureScript and rendering the audio in-browser using the
Web Audio API.
I’ll be documenting my work on this project in a series of upcoming posts.
After coming across being convinced by this article that Docker is a useful tool for running a dev environemnt, I’ve been running my recent Node.js-based FitBit logger project from a Docker container. I’m looking at Amazon’s EC2 Container Service to host the thing, so I’m writing basic Docker rather than something fancier like Compose.
Setting up a Node.js container
This tutorial on the docs.docker.com site goes through the process of building a Docker image for Node.js. Here is the one I ended up with by following their instructions:
FROM centos:6
RUN rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
RUN yum install -y npm
# App
ADD . /src
WORKDIR /src
RUN npm install
# For Dev
RUN npm install -g nodemon
EXPOSE 80
CMD ["node", "/src/server.js"]
Note the install -g nodemon: Nodemon will monitor your filesystem for changes and restart a Node.js server when source code is modified.
This is how I start up my container using Nodemon:
One thing to watch is that you have to have run npm install at least once locally, so that the node_modules directory is created. You need this because the contents of the local source directory will get copied over the container source directory with the -v /Users/miller/dev/fitbit-logger:/src portion of the command.
Now you can edit your code and have it reflected in your Docker container automatically, without having to run additional commands on the container!
For a long time I’ve been working on an animation system built on the Novation Launchpad MIDI controller, and I’ve settled on Clojure as the platform. The data-oriented nature of Clojure seems ideal for representing streams of LED frames and their transformation.
Since I’d already written two interfaces for the Launchpad (for Node.js and Max), I was not eager to go through the same exercise with Clojure. Instead, I decided to run the Launchpad for Max/MSP, and have Clojure send messages to it.
Since the project is an animation system that will be sending a series of frames to the Launchpad (via Max), timing is important, while guaranteeing arrival of all information is not so much. Open Sound Control (OSC) is a real-time data format developed as a modern successor to MIDI. Since it runs on the UDP protocol and is natively supported by Max, it seemed like an easy solution. The osc-clj module (part of the Overtone project) appears to be the main contender for OSC support in Clojure.
Clojure to Max
It’s pretty easy to connect osc-clj to Max. In Max we can receive OSC messages via the udpreceive object. It takes one argument for the port on which you want it to run. I feel like this objects should be called oscreceive, but that’s neither Arthur nor Matha as my mom says.
Max will print any OSC messages it gets on port 1337. Doesn’t get much easier than that.
Let’s create an OSC client in Clojure:
(use 'overtone.osc)
(def c (osc-client "localhost" 1337))
Then say ‘hello’ to Max (send a string and a series of numbers):
(osc-send c "/max" "Hey buddy." (Integer. 1) (Float. 2.7))
Take a look in the Max console - you will see a message like:
received: /max Hey buddy. 1 2.7
So messages are coming through. Note that for numbers, I am creating Java Integer and Float classes to tell osc-clj what OSC data type I want to send. If you use bare Clojure numbers it doesn’t currently seem to properly infer the type.
Max to Clojure
To get messages going in the other direction, we’ll use the udpreceive object’s counterpart udpsend to send data over port 7331.
Lately I’m fascinated with ElasticSearch, and I’d like to start indexing / analyzing the content I’ve built up in my gratefulness journal (thx-journal.net) in the many years I’ve been using it. I decided this would be a good excuse to get familiar with Docker at the same time.
After familiarizing myself with the basics of Docker, I found a project called dockerfile/elasticsearch, a nice image that runs ElasticSearch with an EC2 discovery plugin enabled.
There was one thing small thing missing that I love to have in ElasticSearch: the Marvel plugin. This provides pretty-looking analytics for your cluster, as well
an excellent json developer console (formerly known as Sense). Getting this included in the image was a one liner in the
:
RUN cd elasticsearch && bin/plugin -install elasticsearch/elasticsearch-cloud-aws/2.4.1 && bin/plugin -install elasticsearch/marvel/latest
Ran the new image, and everything just worked (once I figured out how to get my boot2docker VM ip)! Here’s the image on DockerHub: millerpeterson/elasticsearch-marvel-ec2.
Stay tuned for broadcasts regarding functional programming in Clojure, Max/Msp / Clojure interactions, song lyrics data analytics, and the Novation Launchpad.
