Introduction

This is the final part of our Unity+FMOD+GVR tutorial series and we are covering the implementation into Unity.  The videos are separated into two parts, implementing ambience and implementing the gunshot interaction. We hope these tutorials are helpful for you and please contact us at [email protected] if you have questions or comments.   Note that GVR has been rebranded by Google as Resonance Audio. Thanks for watching!

If you missed the first two articles in this series, you can find them here:

Part 1 – Introduction to Spatial Audio

Part 2 – FMOD Setup

 

In the previous video we set up an FMOD project with the GVR plugins. In this video we are going to set up the Unity project for VR and implement the ambience events. I’m building this using a Vive, but GVR and FMOD will work with any VR device and since the SteamVR prefabs use OpenVR, you can use any headset that supports OpenVR, including an Oculus.

Download the FMOD and Unity Projects

You can download the completed Unity Project and FMOD project used in these tutorials here:

DOWNLOADS

Setup

If you are starting from scratch, the first thing we need to do is download the viking village project and the steamVR prefabs (If you are using the finished project this is already done.)  Go to the Unity Asset Store and download them.  Search “steamvr” for the steam VR package, download and import it. Then search “viking unity” and download and import the viking village project from Unity.

If the API Updater give you a warning, click “I made a back up, go ahead” (the version on the store is the backup). When the project loads Steam VR will pop up with a window select “accept all,” this ensures that the project is set up and optimized for VR. 

Next, you will need the FMOD studio Unity integration.  www.fmod.com/downloads. You will need the same version of the Unity integration as the version of FMOD studio you are using. The integration includes the necessary GVR assets that FMOD needs. Import it by going to Assets->Import Package->Custom Package… and browse to where you downloaded the integration.  Then go to FMOD->Event Settings and in project path, and browse to your FMOD project. Also, in the Event settings, scroll down to plugins and add “gvraudio”.

Editing the Level

Open the viking village scene in Assets-> Scenes.

If you start seeing warnings relating to lighting in your console. Go to Window->lighting->settings-> and click generate. You will see a blue bar in the bottom corner of the screen showing the progress of the light maps. It is ok to continue working on the level as this is happening. (I didn’t in the video because I was screen recording and didn’t want to overuse the GPU).

I’m going to enable world boundaries, delete the canvas, event system, mouse lock and quality manager, this holds the previous player controller. Next I’m going to import the SteamVR prefab in SteamVR-> prefabs.  Import the steamvr and the camera rig prefabs. The camera rig is a basic setup that links into the Vive that we are going to extend for our purposes.

Next up, we need to specify our FMOD listener go into the hierarchy and open up the camera rig and camera(head) and open the ears in the inspector. Disable the unity audio listener and click “add component” and type in FMOD listener, then select the script.

Adding the Ambience Objects

Now we are going to add our ambience. Right click in the hierarchy and create an empty gameobject, name it ambience, and add the FMOD studio event emitter, set the play and stop to object start and destroy, respectively, and then select your event from the FMOD event browser.

Now lets test our level! Hit the play button, and we should be able to turn and hear the soundfield changing. Its important to note that it doesn’t matter where in the scene this soundfield is because sound fields do not take into account location; only rotation.

 

Water

We want to set up some more sources so that the ambience changes as we move throughout the level. So lets add another object and add the FMOD emitter for the ocean emitters. Again set the play and stop to start and destroy, then add our event.  On this one it is important to place it accurately, so I’m going to put it  below the listener and close to the shore.  Next, we’ll make a prefab out of it by right clicking in the project window and right-click->create->prefab, then drag the object from the hierarchy to the new prefab. Now drag the new prefab into the scene line the shore with a few more emitters. I’m going to add a new empty game object and place all of the water sounds in there, just to keep my hierarchy cleaner.

Torches

There are quite a few torches throughout the level that we are going to add sounds to, luckily they are prefabs, so we only need to edit one and then apply the changes to all of them. Select one of the torches and go the the prop_torch_01 object and add an FMOD emitter and again link the event and start destroy for start and stop.

Its important in vr to be very precise as to where you are placing your sound emitters. In 3d games everything is playing on a single plane, so if we were designing a car the wheels driving over dirt and the engine could come from the same source and sound fine, but in vr the wheels need a separate source from the engine because in reality those 2 sounds are coming from 2 different places in space. Likewise with a giant monster – in vr you shouldn’t place the foot falls at the same place the roar comes from.

Back to the torch, once you have the source placed on the torch you can go to the pf_torch_stick – the top level of the prefab and hit apply. It will add our changes to every prefab of this object – it will apply ALL changes except for the position and rotation of the transform so be careful when editing. There are 2 other torches in the level that weren’t updated, select one of them and add the event.

Cranes

We also have a sound for the giant cranes. I’m going to add this event to the wheel object and the sound will come from the center of the wheel. Again update the prefab and it will add our sound to the other 2 cranes.

Frogs

I’m also going to add a few frogs in the grass to the level, make a new empty gameobject and add the fmod emitter link up the sound.  I’m going to duplicate this object (ctrl+d) and change it to the other frog sound and move it to another patch of grass. I’m not creating a prefab of this because I only want one of each, but if I wanted more I would make a prefab for both.

Bird

The last piece of ambience we have is a bird.  We’re going to animate the position of the bird so it is flying around the area.  Create a cube, and I’m going to put mine on the wall to start. Add component-> animator and add the FMOD emitter. In our project create an animator controller, name it bird and link it to our bird object. Open the animator window right click and add an event. Then in the project add an animation, link it to the event and make sure to set it to loop in the inspector.  Open the animation window and select the bird. In the animation window add property->transform->position. I’m going to make my animation 1 minute, so scroll wheel to zoom out and move the end keyframe to 60:00. Move the time bar and then set the position.  I’m going to have the bird fly around our head.

Adding Movement

Save the scene and test it out. Be sure to turn off the record mode in the animation window.

 For the last part of this video, we’re going to add a way to move through the level. On the camera rig, I’m going to put it on the left controller, go to the controller left object – it should have the steamvr_tracked object on it already and add 2 components, the steam teleporter script and and steam laser pointer. Change the color to red and the teleport on click to true.

Part 2 – Adding the Gun Interaction

 

In order to add the gun interaction we need to get a gun model.  Go to the Asset store, and to get the one I used, search “gun 1911”. Download and import the package.  Then, add one of the models in the package the right controller and adjust the transform, so it is accurate to the controller. Then turn off the controller.

 

 

Add a line renderer, uncheck world space, set the width to .1, and the positions to (0, 0.7, 0) and 0, 0.7, 100).  This will create a line from the gun barrel to 100 meters in from the of the gun barrel. In order to get the line to be red, we will create a new Material.  In the project right-click->create->Material and set the albedo to red and emission to red. Add the material to the renderer.

Coding the Gun

Create a new C# script, I called it Controller_Gun and attach it to the right controller object.  In your code editor (I used Visual Studio) delete the start and update functions and remove the Monobehaviour parent class.  Replace it with SteamVR_TrackedController.  We want to extend the OnTriggerClicked function to do something when the controller trigger is pressed.  In this script we will also need to add an FMOD event reference and play a oneshot in the OnTriggerClicked function.

Back in the Inspector, link up the FMOD fire event. save and test it out.

We are going to add another script, called Target, that will be placed on everything that can be shot with the gun.  This one will be a MonoBehaviour, be again we don’t need Update or Start.  Add an FMOD Event Ref and create a new function called PlayHitSound that takes a vector3 and it will play an FMOD OneShot at the given location. The objects that will have this script will need a collider and the appropriate FMOD event linked to the script.  I started with the terrain_near (it has a mesh collider) and linked the dirt hit event from FMOD.

Back in the Controller_Gun script we need to add the interaction by using a ray cast and get data from what we hit.  If the object we hit has a Target script, we call the PlayHitSound script on it to play the appropriate hit sound.

Make sure to save and test at this point.

Making the environment responsive

The next part consists of going through each object you want to interact with the gun and making sure it has a collider and a Target script with the appropriate sound. Much of the level has been made with prefabs and is easy to add to one and apply changes to all.  You can also multi-select and add component to add the same component to all selected items. (this is already done in the completed project available for download above)

Adding GVR Audio Rooms

 

The last thing we are going to cover is adding reverb zones. I added 2 to the level: one in the big storage area and one in the blacksmith. In order to do this add a new empty child object to the building you want to have reverb and add the fmod gvr audio room component, then approximate the size of the room.  Any sound emitted in this zone will have reverb applied based on the parameters set in the inspector. In Unity front wall is positive Z, back wall is negative Z, left wall  is negative X, right wall is positive X, ceiling is positive Y and floor is negative Y.

 

Thats it, you now have a interactive VR level! Thanks for watching! again, if you have questions or comments contact us at [email protected].

Melbourne, Australia  (Originally published on Gamasutra)

“Good sound design is more important than ever and increasingly developers rely on the creative talents and specialist technical knowledge of experts to not only source original audio content for their projects, but to take full responsibility for the implementation,” noted Matthew Connor, General Manager of Firelight Technologies. “For producers, this means better outcomes with less risk.”

The dedicated team of sound designers, composers, VO producers, and implementation experts at Somatone have worked on over a thousand game titles, across the Mobile, Desktop, Console, Arcade, and Virtual Reality (VR) platforms.  Close cooperation with Firelight Technologies will deliver premium access to FMOD product knowledge and technical support to ensure their customers enjoy the full benefit of FMOD during development and operation.

“FMOD is a great solution for implementing adaptive audio and gives us the ability to offer our customers a complete service from audio design through mixing to final testing,” said Dale Crowley, Creative Director of Somatone Interactive.  “It is not enough to just create great sounds.  We bring them to life with FMOD, making the audio experience truly interactive. With close support from the amazing team at Firelight, we are getting even better results and our customers love it!”

About Firelight Technologies

 

Firelight Technologies is the developer of FMOD, a complete audio solution for interactive entertainment, supporting all the major console, desktop and mobile platforms. Use FMOD Studio to design adaptive sound and music for games, virtual reality experiences, gaming and simulations.  FMOD Studio integrates easily with popular game engines such as Unity and Unreal and is built on the industry standard FMOD engine which has been used in thousands of games during the last 15 years. For more information visit:www.fmod.com

 

About Somatone Interactive

Established in 2003, Somatone Interactive produces audio content and technology solutions for interactive media.  Widely recognized as one of the video game industry’s most experienced and trusted creative partners, Somatone serves as a leading global provider of original music, sound design, VO, mixing, and audio implementation.  With more than 1,000 titles to its credit spanning all genres, including many top-sellers and tens of millions of daily active users, Somatone maintains longstanding relationships with major publishers, developers, and indies.  Distinguished by years of combined industry experience and a driving desire to deliver world class game audio, the Somatone team includes core staff members with executive experience at major game makers, and cutting-edge sound design, music, and technology professionals.  Visit: www.somatone.com

We are honored to be working with Firelight to bring you a series of feature and tutorial videos for the new release of FMOD 1.10.

It is packed with some fantastic new capabilities, broadly grouped under three categories:  Live Update, Per Platform Optimizations, and Smart Presets.  These greatly increase the speed and flexibility of working in FMOD, and one of my favorites is the new Live Update feature that allows you to swap out audio files without requiring a new build of the game!

I call this Hot-swapping, and the scenario where this is most useful is when working with clients who send us builds of the game with Live Update.  We can test out different sounds realtime, while the game is running, without needing to ask the developer for a new build.  This feature alone makes the update to 1.10 a no brainer, but with all the other improvements and features, it seems like this is one of the most important updates to FMOD Studio in a long time.

To Download FMOD 1.10 for free, simply head over to the FMOD Website.

Here is the introduction and overview of the videos that follow:

And here is the first video, which covers the new capabilities of Smart Presets

Large projects can have thousands of events, creating a huge challenge for developers.  Smart presets provide an innovative workflow solution to managing parameters, effects, and automation across events in you project. This feature greatly improves scalability while also reducing the project’s memory footprint. Smart Presets allow you to link any number of events and have changes propagate across your project.

Smart Presets make it easier to:

• Manage large projects
• Reduce memory footprint
• Edit once and change everywhere
• Provide a dynamic and scalable workflow that is fast and easy to iterate on.

Live Update

Live Update allows you to create audio content and monitor the results while connected to your game in real time. You can mix, iterate, test and optimize your entire project in real-time with instant feedback. Interactive media provides a unique challenge for sound designers. Live update provides a powerful workflow to improve the audio design of your game.

Live Update allows you to:

• Create, Mix, Test, and Optimize in-game
• It allows for fast iteration, and fast results
• All in a workflow designed for games.

Per Platform Optimizations

Getting the audio right for each platform requires an understanding of the key features and limitations each platform, such as: what audio codecs are available, is there a limit to download size, how much RAM is available, how powerful is the CPU, and other questions that you need to understand to deliver the best audio experience you can. FMOD studio makes it easy to develop and optimize by making one project and exporting to multiple platforms. In the project we can set encoding formats and loading strategy for each platform and, new in 1.10, we can set which content and DSP plays on which platform.

The general range of platforms runs from Desktop, with the least limitations, to Mobile, with the most limitations.  On a desktop you generally have lots of memory, a powerful CPU, and unlimited download size. Whereas on mobile, you have limited memory, a less powerful CPU, a small download size and the added limitation of battery capacity. These limitations affect how we design and implement audio for each platform.

FMOD’s platform optimization tools allow us to:

• Have a single project for all platforms
• Have centralized management and testing environment
• Simplify the development and maintenance of a project
• And makes it easy to reuse content on multiple platforms

For more info go to the FMOD Website.

 

Introduction

Welcome to the first of a three part series of in depth tutorials about the art and technology of creating audio for virtual reality.  In this first part, we will be giving a general overview and talking about some of the key concepts of audio in 360 degree surround.  In the following articles, we will go into the details of setting up that audio in FMOD, and then integrating that audio into Unity.  We don’t leave out any of the steps, and even include the FMOD and Unity packages for you to play around with!

In this series of articles, we have created videos that demonstrate the pipeline for working with the Google VR (GVR) audio plugins in FMOD and Unity. We will be using the Viking Village Demo from Unity and a Vive demonstrate, but FMOD and GVR will work with any device.  This video is a collaboration between FMOD, Google and Somatone we would like express our gratitude to everyone who helped to make these videos possible.

 

 

VR Audio Concepts

There are a few concepts and I want to go over before we dive in, which I cover in this video.

 

The first concept is Ambisonics. Ambisonics is the technique that describes creating the full-sphere of sound around a listener; this includes both the stereo panning of the horizontal plane as well as the up and down of the vertical plane.  We will be using First Order Ambisonic files. these are 4 channel files in which the channels do not correspond to speakers, and for the purposes of VR, should be decoded to a stereo output and heard in headphones.

GVR in FMOD

The next concept is the different GVR plugins. There are 3 plugins in FMOD: the listener, source, and soundfield.

• The Listener takes information from both the source and soundfield and applies binaural spatialization and room effects, then converts the information to a stereo output. This should put on the master bus in FMOD.
• The Source takes mono sound files and applies distance attenuation and directivity patterns, then sends that information to the Listener.
• The Soundfield decodes first order ambisonic files based on rotation and sends it to the Listener.

GVR Effects

Spatial Audio in GVR is comprised of 3 effects:

• Interaural time differences – This is the difference in time from when one ear a sound to when the other hears that same sound.
• Interaural level differences – This is the difference in volume between one ear and the other.

• Spectral filtering – This is the changes our outer ear makes to a sound depending on the angle in which it reflects off of the outer ear and into our inner ear. This is the primary way we hear elevation of sounds.

These effects that our ears do for us in the real world must be recreated in a virtual world.

Based on these effects Google has created HRTFs or head related transfer functions that recreate the effects and packaged them into the GVR audio plugins so we can spatialize sounds in virtual reality. The plugins also include distance attenuation curves and can link to the GVR audio room which will simulate early and late reflections (or reverb) in real time.

Additional GVR Settings

The plugins also have settings for directivity patterns, occlusion and spread.

•  The directivity is the way the sound propagates from a source, these work like virtual microphones and mimic the common pickup patterns patterns: omni, cardioid and figure eight. It also includes a sharpness factor – which is the width of the directivity pattern.
• The occlusion factor simulates an object blocking the direct sound from a source and affects high and low frequencies differently and mimics how occlusion works in the reality.
• The spread describes the point the sound comes from. Sounds like gunshots and ricochets can come from small point sources and have no spread, but water sounds will come from wider sources and have higher spread.

Final Thoughts – Key Concepts in VR Audio

Lastly, the videos above cover a few general rules for designing sound in a VR space to make your world seem more believable.

• Use mono source files, unless it is an ambisonic sound field or it is not going to be spatialized (ex: non-diagetic music, internal player VO).
• Only use reverb in enclosed spaces
• Place audio sources as accurately as possible (voice sources should come from the mouth, not the center of the body)
• Use complex waveforms covering a wide spectrum – fundamental waveforms like sine waves do not exist naturally and do not spatialize well.
• If you want a listener to locate a source, play a sound multiple times, make sure to include higher frequencies as they are easier for our ears to pinpoint, and animate it, people have an easier time locating moving sources as opposed to stationary sources.

Stay Tuned…

We will be releasing the remaining  parts of this tutorial series over the next few weeks.  In the next installment, we go into the FMOD setup for different types of VR audio and how to use the GVR plugin.  In the final two installments, we explain in detail how to integrate FMOD into Unity, including the projects themselves and the code required to get some really cool audio hooked up.

 

By Eric Van Amerongen, Sr. Sound Designer, Somatone Interactive

Audio for games: It’s what we do. It’s what we breathe day in and day out here at Somatone. If you’re a producer and want your game to be engaging, creative, cohesive, competitive on the market and most importantly, FUN! for your user’s experience, then factoring in great audio is an important component of your production. From our experience, great audio not only begins with its inception, but also continues through to its final implementation, which is the most important part of game audio production. (more…)

As published recently in Audio Pro International

Sci-fi weapon sound design is the subject for the next installment of our Game Audio column, in partnership with Somatone Interactive.

This month, executive director Michael Bross and senior sound designer Eric Van Amerongen team up to share their insights into the creative process for this particular gaming genre from their respective positions. (more…)