Live Streaming Platform As An Instrument, Experience And Pastime

Harshroom was a prototype to visualize the concept of having a labyrith of rooms containing spatialized audio streams. I used THREE.js for the 3D part and React for the UI. Although the core idea was to not have a visual aspect, it served as a way to communicate and imagine what it could be.

Noisecollab was a little experiment attempting to explore the "collaborative audio" part of the concept. The idea was that participants would record a second of audio, which would then upload to the server. The audio snippet would be inserted into a 10 second audio loop. The updated loop will then prompt all participants to reload their loop, and subsequently all future requests would pull the newest loop.

With Drawcollab, I shifted gears back to graphics and tried building a collaborative drawing experiment. It was mostly to learn Go for server-side code, Redis to manage data, and C++ for OpenGL-based rendering.

Towards the end of 2019, I began to focus back towards audio and the realisation of the original vision. It started with learning about synthesizers, both physical and virtual, and building a self-hosted personal radio stream from my library. I had visited Geoff Stern Art Space and was exposed to the idea of using multi-channel audio for sound art. The synthesis was to imagine taking a collection of many streaming URLs, and feeding each one into a speaker. The speakers would immerse the participant in sound and they could move through the soundscape via streams moving into and throughout the channels.

I think this would work by using a cheap SBC like a Raspberry Pi Zero and connecting it to only one or two channels, and coordinating every SBC used through a local network. This project stalled because of the complexity of trying to use a single machine, as well as handling both input and output, and the requirement of so much hardware and technical setup. I abandoned this approach for something easier for participants to access and use.

After I realised Raspberry Pi + multiple audio interfaces + Icecast + Python + bash wasn't going to work as a teachable system, I moved on to trying to build this as a web app. The experience with that other system was informative and provided me with a reference, but in comparison it felt much harder to achieve the design I had converged on.

At this stage I had departed entirely from the "surrounded by speakers on streaming audio" model and was building a glitchy performance tool. In retrospect, I think I should've spent more time trying to really figure out the validity of the design as something other people would enjoy using. However, I was invited to present my project at workshops and get grants, so it resonated with other people to that extent. However, it was difficult to articulate the coherency between the original concept and what I was now building.

Working in the browser and centralizing the streams came with its own complexity. What my code examples here don't reveal was how hard it was to figure out how to get audio to segment then seamlessly join. Decompressed audio doesn't look like a perfect representation of a pre-compressed chunk of audio. Each type of compression results in a different looking artefact as well. The wave is shaped different, if you treat each segment as an isolated unit you will have compression artifacts, there might be padding, and all of it has to be blended in with the rest of the audio to make it not pop. Beyond the audio you have to handle the load on the server, storage, browser API support, and performance.

This is meant to be an image of a desktop computer, a laptop, a phone, a server, and human ears, all tangled together through a spaghetti-like network.

After the first couple of workshops I ran with _noisecrypt, I wanted to make some improvements of the system. The UI wasn't quite where I wanted it to be in terms of how usable it was, there was a lot of copy and paste, separate web apps for each part, and it wasn't clear how to use it altogether. I wanted to be able to improve some of the client-side parts separately to the web app, so I moved them into separate modules. I changed the same to try to indicate the type of system it had become as "noisecrypt" I felt was invoking a different mental image of the system to what it was.

One experiment that I couldn't get working was the use of Audioworklets. They were fairly new to browsers (added April 2021), and they were meant to replace the deprecated ScriptProcessorNode. I was new to how to optimize for them, so I couldn't figure out how to manage the very small buffer they handled. I was doing too much work in them, which would cause overruns, and therefore pops in the audio stream. I eventually abandoned them and returned to the deprecated API.

I also spent a lot of effort building out a completely custom UI system and the framework (on top of redom) to go with it. I probably should've avoided that, but I was really interested in how to think through the problems and find solutions (preferably, simpler) to those the mainstream frameworks solved.

My performance gear used as part of an arts grant I received for this project.

After the performance grant and conference paper, I felt like I had learned what I intended to and more. The project itself diverged so far from the original vision as to no longer be recognisable. It went from conceptually a live audio virtual world to a glitchy aleatoric networked feedback web music performance platform.

It was fun to use and incredibly valuable for my learning experience. The reason for the divergence was the practicality and feasibility of building out the vision and my pursuit of the different tangents generated along the way. In the end I had other projects take priority and wanted to learn something new.