Just as with a film or video game, the importance of sound in virtual reality is often unappreciated. People typically do not realize this until they experience content that has a soundtrack, but is turned off. Visual elements tell the story, but sound is just as important in virtual reality because it affirms whether or not the experience is believable, and it creates emotional impact.
Another way to understand the importance of sound in virtual reality is to measure how much companies are investing in audio technology. For example, Facebook and YouTube are both strongly committed to supporting VR content on their sites. For this purpose, the audio team at Facebook has developed a set of free audio tools for VR production called Spatial Workstation, which is intended to empower the production of virtual reality audio.
The VR Sound is All Around Your Head
Virtual Reality adds a new dimension — literally — to the experience of a narrative through immersion. One has a sense of “presence” in the scene, of personal participation. From a visual standpoint, it means that the VR participant feels a part of the scene and is no longer invisible to the characters in the story.
In a recent VR film by Baobab Studios called “Invasion,” the main character (a rabbit) witnesses the landing of an alien spacecraft. The aliens see the rabbit and begin a menacing approach. At one point, the rabbit moves behind the spectator, so that the spectator is positioned between the rabbit and the aliens. One can feel in a very visceral way the vulnerability of being in the middle of a threatening situation.
What does this vivid presence mean for sound? We have a very highly developed and precise sense of sound localization, which means we can tell with considerable accuracy where a sound comes from. The way that we do this is based on how our brains process the change in a sound as it travels around our heads. The subtle color differences between the sound that enters our two ears by different paths around our heads are processed as spatial location information. Researchers have measured the effect of different head shapes and sizes on how the ear receives a sound from a point in space — which is known as a “head-related transfer function” or HRTF for short. This HRTF data can then be applied to individual sounds so that they are wrapped in spatial cues. Binaural audio only works with headphones, but that is perfect for VR where the user is isolated from the natural world within a headset.
The More Things Change, the More They Stay the Same
In the early days of silent movies, music was performed live and served as commentary for the cinematic narrative’s action and flow. Then, music and sound started being recorded and printed onto film. As the technology of film sound became more sophisticated, encompassing multiple channels of sound and a corresponding number of speakers placed around the theater, sound became a critical factor in the success of the film delivering its story.
Innovations such as Dolby Noise Reduction, surround sound, the addition of a sub-bass channel and digital recording media have all contributed enormously to an increasingly immersive cinematic experience. As soundtrack production has evolved into even more immersive applications such as videogames and now VR, audio technology has reached a very high level of technical perfection. In terms of clarity and fidelity, fullness and richness, spatial precision, dynamic range and power, current audio technology is a crowning achievement of engineering and innovation. Yet in spite of this technical progress, the role of a soundtrack has changed little from the beginning. The purpose of audio in a film boils down to two things. The first is sound design, which includes sound effects and audible environmental factors such as wind and street noise, and enhances the believability of the narrative. The second is music, which delivers emotional impact. And so it is with VR, which imposes greater technical requirements on a soundtrack than before, but relies on sound to make the situation believable and to deliver emotional impact. Like cinema before it, some VR creators claim that sound makes up 50 percent of the VR experience.
An Uncertain Future for Virtual Reality
No one knows how VR will develop, what direction it will take or which current media roles it will supplant. It’s still too new. There is significant VR development already underway for video games, education, military, news and sports casting, narrative film, music videos and much more.
As VR evolves and develops, it will challenge audio production in new ways, spurring innovation. VR has already given new life to audio technologies such as Ambisonics and binaural recording, audio systems previously sidelined for lack of sufficient application. As always, the role of audio in virtual reality might not change significantly, but the way that users experience it might be expanded and extended in surprising ways in the near future.
