This was originally posted by KaoruAshihara
Let me introduce some panoramic videos with panoramic sounds.
You need stereo headphones to listen to the sound.
In the mobile devices, not only the image but the sound follows pan and tilt of the device.
https://unit.aist.go.jp/hiri/hi-infodesign/as_ss0/railway.html
https://unit.aist.go.jp/hiri/hi-infodesign/as_ss0/sidewalk.html
https://unit.aist.go.jp/hiri/hi-infodesign/as_ss0/lost_species.html
They are from the site below where you can see more examples.
These videos (not include sound) were captured with THETA m15.
The information below is from the SOPA site
Outline
In real life, localization of sound sources are rarely static. When listeners turn their head to the left or right in the real-world environment, the flow of sound from their viewpoint instantly changes. Is it possible to transmit such sort of sensation online?
Our answer is, ‘Yes it is.’
On this site, we introduce our technology for online transmission of spatial audio information. This technology is named ‘SOPA’ which stands for ‘Stream Of Panoramic Audio’ and its features can be summarized as follows.
- Miniature head simulator system — Portable device to capture spatial sounds
- Panning on demand — Interactive manipulation of panning
- Virtual cardioid microphone — Programmatically induced directionality
- Low computational cost
- System with high expandability
- You can create and share panoramic sounds
Miniature head simulator system
Spatial audio information can be captured by a device called ’ Miniature Head Simulator.’ Since it is far smaller than the conventional dummy head, the miniature head simulator provides us with a convenient tool for capturing spatial sound.
Panning on demand
A unique feature of this technology is that it allows for interactive manipulation of panning.
In real-world situations, the locations of sound sources vary according to the movement of the listener’s head. If a listener directly facing the sound source, for instance, a loudspeaker, turns his or her head to the left or right, the sound no longer arrives from the front. Therefore head movements can be interpreted as movements of the world from the viewpoint of the listener. In reality, this never occurs when listening to music through headphones since the headphones rotate along with the head.
SOPA makes it possible to control the panning of the sound during its reproduction.
Virtual cardioid microphone
SOPA data contain not only an audio data stream but also its spatial information. Spatial information can be used to produce directionality so that the listener can pick up sounds from any desired direction and reject sound from other directions.
The listener can control the target direction while listening to the sound.
More about the virtual cardioid microphone
Low computational cost
Spatial audio information captured by the miniature head simulator system can be encoded to a custom data format. Even though its size is euivalent to the conventional stereo WAV file, a SOPA file contains not only the acoustic data stream but also the spatial information.
SOPA data can be decoded by using only a simple trigonometric function and Fast Fourier Transform. Since this process does not require much computer resources, a SOPA file can be reproduced not only on the desktop computers but also on mobile devices.
By using the SOPA format, spatial audio information can be streamed to the browsers, smartphones, and tablets with a low bit rate.
For instance, a panoramic audio player for iPhone ‘mySopa’ and a panoramic audio player for Android ’ Immerspot’ are available.
Moreover, you can provide a Web site with immersive panoramic sounds by using SopaJS. SopaJS is an open source JavaScript library package for reproducing panoramic sounds on the Web browsers.
High expandability
A SOPA file contains only the monaural audio data stream and the directional information. How to reproduce the data is not specified anywhere in the SOPA file. How to reconstruct the space acoustically from the SOPA file is open to the developers.
Although all demonstrations presented on this site require stereo headphones, spatial audio information can be reconstructed from a SOPA file also for the multi-channel loudspeaker system.
We constructed, for instance, a 6-channel loudspeaker system to reproduce SOPA data that is illustrated in the figure below. The system is named ‘Panoramic Sound Spot System [1].’
Fig. 6-channel Panoramic Sound Spot System
Although this 6-channel loudspeaker system is rather small and for the personal use, the system might be expanded for the theatrical performance.
A Java program for the 6-channel Panoramic Sound Spot System ‘pss6tr’ is available. You can download a jar file from pss6tr.jar (SOPA archives).
A Java program for the 4-channel Panoramic Sound Spot System ‘pss4tr’ is also available. You can download a jar file from pss4tr.jar (SOPA archives).
Demonstration
Here, we provide samples demonstrating the capabilities of SOPA.
In the demonstrations, stereo headphones are necessary.
The demo can be started by clicking the thumbnail below .
Once the image is loaded and the page says “Click to start” or “Tap to start,” you can start reproducing the sound by clicking (tapping) on the image.
If the instruction saying “Tap on the image,” appears, tap the screen and the browser starts loading data. When the necessary data are loaded, the page will say “Tap to start” and you can start reproducing panoramic sound by tapping the image.
During the reproduction, the panning of sound can be controlled by dragging the mouse to the left or the right.
If you are using the mobile device, pan and tilt can be controlled by the rotation of the device.
Sopa_version2_22k.sopa
Sidewalk.sopa
Thanks to Three.js for JavaScript 3D library.
Recommended browsers
Safari is recommended for iOS and Mozilla Firefox is recommended for Android devices.
WebGL has to be supported and enabled by the browser.
The graphic card of your device has to support WebGL.
You can make it
If you are interested in creating panoramic sounds on your own, see Create and share panoramic sound. To make your Web site full of immersive panoramic sounds, consider using the JavaScript library “SopaJS.”
Reference
[1] Motoki Kawano, Kaoru Ashihara, Kosuke Taki, Kazuki Azuma, Shogo Kiryu,“Demonstration of the panoramic sound field system.” Proc. Auditory Res. Meeting, H-2014-83, 2014
Copyright© 2016 AIST