As there has been a lot of great activity in this community on using Unity and the THETA V for live streaming applications, I am posting the README.md from my test project here. This project is still in the early stage, but it may help other people working on similar projects at this very minute.
RICOH THETA V with HTC Vive and SteamVR
Sample project for using RICOH THETA with HTC Vive and SteamVR for telepresence applications.
Display the 4K video stream from a THETA V inside a HTC Vive headset.
The RICOH THETA is the world’s most popular 360 camera. The THETA V model was released in late 2017. The “V” outputs a 4K equirectangular video stream with H264 compression 3840×1920/29.97fps as well as a 2K 1920x960/29.97 stream.
You can easily use a Unity webcam texture to wrap the THETA V 4K video stream onto a sphere to provide a telepresence experience for industrial or experiential applications.
This test application illustrates the following techniques:
- identify THETA V camera inside of Unity
- assign THETA V to a texture
- use flip normals on a Blender sphere to display the webcam texture to the inside of the Sphere. The Blender sphere is imported into Unity
- use negative scale values to solve mirroring problem of objects in sphere appearing in reverse
- display THETA V stream inside of a HTC Vive headset with SteamVR
- track head movements and controller movements with SteamVR
- use THETA V as a microphone and output audio to HTC Vive
- initial configuration tests to orient the headset and the camera to simulate real-world telepresence
These techniques are compiled from the theta360.guide community in the Unity Development discussion conference. If you have any questions or would like to tell us about your project please join this free and independent community of developers. We’d love to hear from you.
Communications gear for the next gen!
Software Used in This Test
Tests were done with the following:
- Unity 2017.4.0f1 (earlier version should work)
RICOH THETA V
- RICOH THETA V firmware 2.11.1 (or higher)
- RICOH THETA V live streaming driver 1.0.1 64 bit - You NEED 1.0.1 or higher to work with Unity
Thanks and Credits
Thanks to the great community of contributors at theta360.guide.
Special thanks to Ricoh for working with the independent development community and updating the THETA V live streaming driver to work with Unity.
joshapplman for the technique of using negative scale to reverse the mirroring effect inside of the sphere
Nima for the technique to use the THETA V microphones with Unity.
Quick Test Without Installing Unity
This is a skeleton project for developers using Unity and is not intended for use as a binary.
If you want to check it out quickly, I have included a binary in the directory
release. If you don’t want to download the entire repo, it may be easier for you to download zipped package from the releases section of GitHub. The release file is 15.2MB.
Extract the file. There should be a Windows 10 64 bit executable with a green THETA logo.
Turn on THETA V in Live Streaming Mode
To simulate the perspective a standing person, put the THETA V on a tripod at eye level. Connect the THETA V with a USB cable and put it into live streaming mode.
Connect HTC Vive to Your Computer
Align Camera and Headset
The HTC Vive default position is pointed in the same direction as the rear THETA V camera lense. The rear lense is the one that is facing away from the shutter button. A person talking directly to the rear lense will appear to be talking to the face of the person wearing the HTC Vive headset.
Start Application on Windows
Double-click on theta_v_htc_vive_test.exe on your Windows 10 computer.
Wait for App to Load
The app should load in less than 5 seconds.
The screen will turn black as the headset adjusts to the THETA V input.
Put Headset On and Enjoy Telepresence
With the default SteamVR features, you can do the following:
- track both controllers inside of the headset. Position is accurate
- take screenshots from inside of the headset using the controller buttons
I am not using the controllers to grab anything right now.
The script will use the first audio device it finds. Adjust this line in the code to properly identify the THETA V. The script will display the connected microphones to the debug console of Unity.
public const int THETA_V_AUDIO_NUMBER = 0;
Update RICOH THETA V Live Streaming Driver
You need 1.0.1 or later. Earlier versions will not work. See this article for info. Note how Ricoh updated the driver is response to our community feedback.
Viewing Inside of Sphere
I am using a flip-normals sphere that I created in blender. This should be included in the package.
If you’re interested in building your own sphere with Blender,
this video will give you a step-by-step process.
The flip-normals sphere causing the scene to appear like a mirror image. To correct this problem, I am inverting the sphere with negative scale. -8, -8, -8.
Original hack by @joshapplman
I needed to rotate the X axis of the SteamVR rig by 180 degrees in order to get it to work.
Specifying the Correct RICOH THETA V driver
The script should automatically detect the webcams on your system. By default, the script will look for
RICOH THETA V 4K. If you want to use a different resolution or a different driver, you will need to modify the script.
The script will also display all webcams attached to your computer for debugging.
Height of THETA V
Put the bottom of the THETA V at eye level for a telepresence experience that is close the real world. If the camera is at eye level, the person you are talking to should be face the lense that is opposite the shutter button.
Alternately, you can change the perspective to a surreal experience. Several people wearing the headset have mentioned that they like it when the THETA V is close to the ground as they feel like a small animal.
The cool thing about developing for the VR headset is that you are forced to stand away from your computer to test it. You may even need your friend to help you with the headset.
Thanks to Ricoh for these cool THETA V t-shirts they gave out at CES.