Dynamically Interactive Audio

VR is all about immersion. Part of the 20perceived realism comes from interacting with the virtual world on the condition that the environment, and all interactable objects, behave as anticipated when compared to reality. You see a whiteboard marker, you expect being able to pick it up. When a game prevents you from doing so, even though the marker is well within reach, it breaks immersion. If you can pick the marker up, you expect this action to make a sound. Even though we don’t really think about it, everyday life is filled with all these small and seemingly unimportant sounds. But they are far from unimportant. These small sound cues tell us that all is well in the world, that physics works as anticipated today as well. If we would pick up a marker one morning in real life and it didn’t make a small scraping sound against the desk, we would be mesmerized at first. Pick it up again and drop it on the floor. Again, without an audible impact this would be highly worrying. Pick up the marker from the floor and throw it really hard into the wall. Let’s say the high-velocity impact did not make an expected louder sound. Now would be a good time to freak out. Over a marker.

This is where our Dynamically Interactive Audio System comes into play.

At Windfall Studios, we firmly believe that immersion is key. This means an environment you can interact with. You should be able to pick things up, drop them or throw them away. As a consequence, the audio needs to reflect this, or else the immersion breaks. Since the game engine platform we use, Unity, does not provide anything to accomplish this we have designed and created our own toolset consisting of custom code, scripting and object oriented systems to weave all of it together.

In World Apart every object, no matter how mundane, always has sounds for interaction. I.e. picking it up, dropping it but also throwing it really hard. I make this possible through interaction scripts that in turn trigger code and scripts to choose the appropriate sound recording to match what just happened. The system also gets game data such as velocity (i.e. exactly how hard the object hits another object or wall, floor and so on). I can set a threshold to trigger particularly loud and noisy impact sounds when this happens, as opposed to merely putting the marker back onto the desk.

Every time you interact with an object in reality, the sound will always be slightly different. Hearing the very same audio recording repeatedly will also break immersion. Due to this I record multiple variations for every object’s sounds and pick out a matching set. A variation that is too different from the other ones will stand out, so this is a matter of balancing how many and how different each recording should be. Randomizing sounds is nothing new in game audio, however this is an important part for the dynamic system to deliver the desired experience.

I record every sound source myself when I can. Sometimes it is very easy to do. Getting some pencils, duct tape rolls, coffee mugs and such more or less identical to the 3D objects in the game to record is straightforward. Squeaky dinosaurs? No problem! Dimension opening portal engines? Not so much. (I could tell you about how I recorded the portal but then, again, TPTB would make me disappear).

At one time I realised that I had just thrown my own personal computer tablet really hard onto a hard floor surface repeatedly, just to get some good recordings. Three times, and it was still intact. That’s when I understood what I was actually doing; Trying to break my own tablet. So I stopped doing that.

This is why, in World Apart, there are only three recorded audio variations on throwing a tablet really hard at the floor.0

Tomas Danko is the Head of Audio at Eutopia Studios, and has been working with audio engineering and music since the early 80’s, earning him a solid reputation as a legendary C64 musician and a highly sought-after sound designer.