HowTo: Use RICOH THETA V with HTC Vive and SteamVR


#1

As there has been a lot of great activity in this community on using Unity and the THETA V for live streaming applications, I am posting the README.md from my test project here. This project is still in the early stage, but it may help other people working on similar projects at this very minute. :slight_smile: :theta:

RICOH THETA V with HTC Vive and SteamVR

Sample project for using RICOH THETA with HTC Vive and SteamVR for telepresence applications.

Display the 4K video stream from a THETA V inside a HTC Vive headset.

Overview

The RICOH THETA is the world’s most popular 360 camera. The THETA V model was released in late 2017. The “V” outputs a 4K equirectangular video stream with H264 compression 3840×1920/29.97fps as well as a 2K 1920x960/29.97 stream.

You can easily use a Unity webcam texture to wrap the THETA V 4K video stream onto a sphere to provide a telepresence experience for industrial or experiential applications.

This test application illustrates the following techniques:

  • identify THETA V camera inside of Unity
  • assign THETA V to a texture
  • use flip normals on a Blender sphere to display the webcam texture to the inside of the Sphere. The Blender sphere is imported into Unity
  • use negative scale values to solve mirroring problem of objects in sphere appearing in reverse
  • display THETA V stream inside of a HTC Vive headset with SteamVR
  • track head movements and controller movements with SteamVR
  • use THETA V as a microphone and output audio to HTC Vive
  • initial configuration tests to orient the headset and the camera to simulate real-world telepresence

These techniques are compiled from the theta360.guide community in the Unity Development discussion conference. If you have any questions or would like to tell us about your project please join this free and independent community of developers. We’d love to hear from you.


Communications gear for the next gen!

Software Used in This Test

Tests were done with the following:

Thanks and Credits

Thanks to the great community of contributors at theta360.guide.

Special thanks to Ricoh for working with the independent development community and updating the THETA V live streaming driver to work with Unity.

This is based on the project by @zimmermegan for the THETA S.

Additional credits:

  • DrustZ for the initial hack of the THETA S registry that led to ZimmerMegan getting Unity to work with the THETA V. This led to Ricoh updating the driver.

  • joshapplman for the technique of using negative scale to reverse the mirroring effect inside of the sphere

  • Nima for the technique to use the THETA V microphones with Unity.

Quick Test Without Installing Unity

This is a skeleton project for developers using Unity and is not intended for use as a binary.
If you want to check it out quickly, I have included a binary in the directory release. If you don’t want to download the entire repo, it may be easier for you to download zipped package from the releases section of GitHub. The release file is 15.2MB.

Extract the file. There should be a Windows 10 64 bit executable with a green THETA logo.

app-binary

Turn on THETA V in Live Streaming Mode

To simulate the perspective a standing person, put the THETA V on a tripod at eye level. Connect the THETA V with a USB cable and put it into live streaming mode.

image

Connect HTC Vive to Your Computer

htc-vive-connection

Align Camera and Headset

The HTC Vive default position is pointed in the same direction as the rear THETA V camera lense. The rear lense is the one that is facing away from the shutter button. A person talking directly to the rear lense will appear to be talking to the face of the person wearing the HTC Vive headset.

theta-v-htc-v-direction

Start SteamVR

steamvr

Start Application on Windows

Double-click on theta_v_htc_vive_test.exe on your Windows 10 computer.

binary-click-me

Wait for App to Load

The app should load in less than 5 seconds.

unity-app

The screen will turn black as the headset adjusts to the THETA V input.

Put Headset On and Enjoy Telepresence

Controllers

With the default SteamVR features, you can do the following:

  • track both controllers inside of the headset. Position is accurate
  • take screenshots from inside of the headset using the controller buttons

I am not using the controllers to grab anything right now.

Audio

The script will use the first audio device it finds. Adjust this line in the code to properly identify the THETA V. The script will display the connected microphones to the debug console of Unity.

public const int THETA_V_AUDIO_NUMBER = 0;   

Update RICOH THETA V Live Streaming Driver

You need 1.0.1 or later. Earlier versions will not work. See this article for info. Note how Ricoh updated the driver is response to our community feedback. :slight_smile:

Viewing Inside of Sphere

I am using a flip-normals sphere that I created in blender. This should be included in the package.

If you’re interested in building your own sphere with Blender,
this video will give you a step-by-step process.

flip-normals

Mirroring

The flip-normals sphere causing the scene to appear like a mirror image. To correct this problem, I am inverting the sphere with negative scale. -8, -8, -8.
Original hack by @joshapplman

sphere-negative-value

Inverted Camera

I needed to rotate the X axis of the SteamVR rig by 180 degrees in order to get it to work.

invert-camera

Specifying the Correct RICOH THETA V driver

The script should automatically detect the webcams on your system. By default, the script will look for RICOH THETA V 4K. If you want to use a different resolution or a different driver, you will need to modify the script.

The script will also display all webcams attached to your computer for debugging.

webcam-detect

Height of THETA V

Put the bottom of the THETA V at eye level for a telepresence experience that is close the real world. If the camera is at eye level, the person you are talking to should be face the lense that is opposite the shutter button.

Alternately, you can change the perspective to a surreal experience. Several people wearing the headset have mentioned that they like it when the THETA V is close to the ground as they feel like a small animal.

Fun

The cool thing about developing for the VR headset is that you are forced to stand away from your computer to test it. You may even need your friend to help you with the headset. :slight_smile:

testing-fun

Thanks to Ricoh for these cool THETA V t-shirts they gave out at CES.

testing-fun2


RICOH THETA VR Lounge Live in New York - June 19-20
RICOH THETA V Plug-in Microconference - San Francisco - Thurs, Aug 30, evening
THETA S WiFi Streaming with Unity
#2

I updated the project README to indicate that the THETA V audio is single channel in live streaming mode. It is not 4 channel spatial audio for the stream. The 4 channel spatial audio only works with video from files. I have a note out to a friend at Ricoh to check if there’s any possibility of a firmware update on this in the future as I think using spatial audio would improve the telepresence experience.


#3

I uploaded a YouTube video that covers usage of the binary and orientation of the THETA V and HTC Vive headset.

You don’t need to install or use Unity to follow the steps in the video.


#4

Here’s another video showing how you can build the application with Unity, starting from the very beginning of a new project.


#5

New video providing a quick walkthrough of the core script and the three main tips to get the THETA V live stream to display properly in the HTC Vive headset.


#6

Noticing news about “HTC Vive-equipped VR arcades” being built:


#7

Do you also see these types of commercial applications for Oculus Rift (link on amazon) or Oculus Go (link on Amazon to 32GB model)? I’m wondering if it’s coincidence that Dave & Buster’s is using HTC Vive or if HTC Vive (link on Amazon) is generally more widely used for general VR experience platforms.

The two comments below the article are interesting:
image

The personal hygiene concern is common. We saw this at the THETA VR Lounge in Palo Alto.

The solution we’re going to try at DeveloperWeek NY is to use the personal hygiene eye masks below. Please remember to pick them up from me in Palo Alto before you go to New York.

The second comment by TryBe seems to indicate a preference for HTC Vive gear.

Remember that GITAI founder Sho Nakanose was also using an HTC Vive.

As we’re discussing buying an Oculus Go,


#8

After disucssions with @jcasman about the appearance of Mothra in the new upcoming Godzilla movie, I decided to put moths into the HTC Vive demo. I also updated the lighting source.

The updated moth demo unity package is here

Windows 10 binary is here. Requires SteamVR to run.

@jcasman for the demo at SIGGRAPH, I think the problem with intermittent dropped video feed might be related to the headset sync getting blocked if the base stations do not have a clear path to the headset.


#9

I had a great time at SIGGRAPH in Vancouver. Met over a 150 people at the booth and demoed the live streaming THETA V in an HTC Vive headset to at least 60-70 people.

Lots of VFX people who used the THETA “every day,” especially to quickly grab HDRi data for use on site for broadcast or CG development. Lighting, reflections and more. The THETA is super portable, low-cost, and can do bracketing of multiple images that are then combined in Photoshop. Really cool to meet so many people using the THETA so heavily.

Posting just a few photos from SIGGRAPH:

Headed to the show, grabbing a quick snack and charging up my THETA in SFO:

Busy booth, TK Tone, THETA product manager, doing multiple demos:

I’m in Canada, so I have to have at least one meal at Tim Horton’s:


Shoot 360 HDRs for VFX with a Ricoh Theta - By Sam Winkler - Visual Designer
#10

This is great information. I understand about using the THETA to grab HDRI data.

What would they use the 4K 360° live streaming for?

Do they need the live streaming into the HTC Vive headset? Does it serve any type of demo or mockup purpose in the VFX industry?


#11

VFX guys like to capture HDR images with the THETA. They can light CG models using 360 HDRs. They get accurate lighting direction and intensity very quickly this way.

I believe they do not use live streaming at all.


#12

Wow, that’s a bit discouraging. Was there interest in the HTC Vive demo at SIGGRAPH? If there was, what is the possible application? Maybe they can use the live streaming as a form of visual monitoring?

If there wasn’t interest in the HTC Vive demo, we can try and build more examples of HDRI use, which is what I’m trying to do now.

As far as HDRI, there appears to be platforms like Unity and a set of platforms like Maya and Blender. We have some Unity examples to get the lighting information.

I’m thinking of going through this Blender example: