HowTo: Use RICOH THETA V with HTC Vive and SteamVR

[codetricity]

As there has been a lot of great activity in this community on using Unity and the THETA V for live streaming applications, I am posting the README.md from my test project here. This project is still in the early stage, but it may help other people working on similar projects at this very minute.

RICOH THETA V with HTC Vive and SteamVR

Sample project for using RICOH THETA with HTC Vive and SteamVR for telepresence applications.

Display the 4K video stream from a THETA V inside a HTC Vive headset.

Overview

The RICOH THETA is the world’s most popular 360 camera. The THETA V model was released in late 2017. The “V” outputs a 4K equirectangular video stream with H264 compression 3840×1920/29.97fps as well as a 2K 1920x960/29.97 stream.

You can easily use a Unity webcam texture to wrap the THETA V 4K video stream onto a sphere to provide a telepresence experience for industrial or experiential applications.

This test application illustrates the following techniques:

• identify THETA V camera inside of Unity
• assign THETA V to a texture
• use flip normals on a Blender sphere to display the webcam texture to the inside of the Sphere. The Blender sphere is imported into Unity
• use negative scale values to solve mirroring problem of objects in sphere appearing in reverse
• display THETA V stream inside of a HTC Vive headset with SteamVR
• track head movements and controller movements with SteamVR
• use THETA V as a microphone and output audio to HTC Vive
• initial configuration tests to orient the headset and the camera to simulate real-world telepresence

These techniques are compiled from the theta360.guide community in the Unity Development discussion conference. If you have any questions or would like to tell us about your project please join this free and independent community of developers. We’d love to hear from you.

Communications gear for the next gen!

Software Used in This Test

Tests were done with the following:

• Unity 2017.4.0f1 (earlier version should work)
  • SteamVR Plugin for Unity
• RICOH THETA V
  • RICOH THETA V firmware 2.11.1 (or higher)
• RICOH THETA V live streaming driver 1.0.1 64 bit - You NEED 1.0.1 or higher to work with Unity

Thanks and Credits

Thanks to the great community of contributors at theta360.guide.

Special thanks to Ricoh for working with the independent development community and updating the THETA V live streaming driver to work with Unity.

This is based on the project by @zimmermegan for the THETA S.

Additional credits:

• DrustZ for the initial hack of the THETA S registry that led to ZimmerMegan getting Unity to work with the THETA V. This led to Ricoh updating the driver.

• joshapplman for the technique of using negative scale to reverse the mirroring effect inside of the sphere

• Nima for the technique to use the THETA V microphones with Unity.

Quick Test Without Installing Unity

This is a skeleton project for developers using Unity and is not intended for use as a binary.
If you want to check it out quickly, I have included a binary in the directory release. If you don’t want to download the entire repo, it may be easier for you to download zipped package from the releases section of GitHub. The release file is 15.2MB.

Extract the file. There should be a Windows 10 64 bit executable with a green THETA logo.

Turn on THETA V in Live Streaming Mode

To simulate the perspective a standing person, put the THETA V on a tripod at eye level. Connect the THETA V with a USB cable and put it into live streaming mode.

Connect HTC Vive to Your Computer

Align Camera and Headset

The HTC Vive default position is pointed in the same direction as the rear THETA V camera lense. The rear lense is the one that is facing away from the shutter button. A person talking directly to the rear lense will appear to be talking to the face of the person wearing the HTC Vive headset.

Start SteamVR

Start Application on Windows

Double-click on theta_v_htc_vive_test.exe on your Windows 10 computer.

Wait for App to Load

The app should load in less than 5 seconds.

The screen will turn black as the headset adjusts to the THETA V input.

Put Headset On and Enjoy Telepresence

Controllers

With the default SteamVR features, you can do the following:

• track both controllers inside of the headset. Position is accurate
• take screenshots from inside of the headset using the controller buttons

I am not using the controllers to grab anything right now.

Audio

The script will use the first audio device it finds. Adjust this line in the code to properly identify the THETA V. The script will display the connected microphones to the debug console of Unity.

public const int THETA_V_AUDIO_NUMBER = 0;   

Update RICOH THETA V Live Streaming Driver

You need 1.0.1 or later. Earlier versions will not work. See this article for info. Note how Ricoh updated the driver is response to our community feedback.

Viewing Inside of Sphere

I am using a flip-normals sphere that I created in blender. This should be included in the package.

If you’re interested in building your own sphere with Blender,
this video will give you a step-by-step process.

Mirroring

The flip-normals sphere causing the scene to appear like a mirror image. To correct this problem, I am inverting the sphere with negative scale. -8, -8, -8.
Original hack by @joshapplman

Inverted Camera

I needed to rotate the X axis of the SteamVR rig by 180 degrees in order to get it to work.

Specifying the Correct RICOH THETA V driver

The script should automatically detect the webcams on your system. By default, the script will look for RICOH THETA V 4K. If you want to use a different resolution or a different driver, you will need to modify the script.

The script will also display all webcams attached to your computer for debugging.

Height of THETA V

Put the bottom of the THETA V at eye level for a telepresence experience that is close the real world. If the camera is at eye level, the person you are talking to should be face the lense that is opposite the shutter button.

Alternately, you can change the perspective to a surreal experience. Several people wearing the headset have mentioned that they like it when the THETA V is close to the ground as they feel like a small animal.

Fun

The cool thing about developing for the VR headset is that you are forced to stand away from your computer to test it. You may even need your friend to help you with the headset.

Thanks to Ricoh for these cool THETA V t-shirts they gave out at CES.

[codetricity]

I updated the project README to indicate that the THETA V audio is single channel in live streaming mode. It is not 4 channel spatial audio for the stream. The 4 channel spatial audio only works with video from files. I have a note out to a friend at Ricoh to check if there’s any possibility of a firmware update on this in the future as I think using spatial audio would improve the telepresence experience.

[codetricity]

I uploaded a YouTube video that covers usage of the binary and orientation of the THETA V and HTC Vive headset.

You don’t need to install or use Unity to follow the steps in the video.

[codetricity]

Here’s another video showing how you can build the application with Unity, starting from the very beginning of a new project.

[codetricity]

New video providing a quick walkthrough of the core script and the three main tips to get the THETA V live stream to display properly in the HTC Vive headset.

[jcasman]

Noticing news about “HTC Vive-equipped VR arcades” being built:

Dave & Buster’s is building HTC Vive-equipped VR arcades

[codetricity]

Do you also see these types of commercial applications for Oculus Rift (link on amazon) or Oculus Go (link on Amazon to 32GB model)? I’m wondering if it’s coincidence that Dave & Buster’s is using HTC Vive or if HTC Vive (link on Amazon) is generally more widely used for general VR experience platforms.

The two comments below the article are interesting:

The personal hygiene concern is common. We saw this at the THETA VR Lounge in Palo Alto.

The solution we’re going to try at DeveloperWeek NY is to use the personal hygiene eye masks below. Please remember to pick them up from me in Palo Alto before you go to New York.

https://amzn.to/2kPtxgU

The second comment by TryBe seems to indicate a preference for HTC Vive gear.

Remember that GITAI founder Sho Nakanose was also using an HTC Vive.

As we’re discussing buying an Oculus Go,

[codetricity]

After disucssions with @jcasman about the appearance of Mothra in the new upcoming Godzilla movie, I decided to put moths into the HTC Vive demo. I also updated the lighting source.

The updated moth demo unity package is here

Windows 10 binary is here. Requires SteamVR to run.

@jcasman for the demo at SIGGRAPH, I think the problem with intermittent dropped video feed might be related to the headset sync getting blocked if the base stations do not have a clear path to the headset.

[jcasman]

I had a great time at SIGGRAPH in Vancouver. Met over a 150 people at the booth and demoed the live streaming THETA V in an HTC Vive headset to at least 60-70 people.

Lots of VFX people who used the THETA “every day,” especially to quickly grab HDRi data for use on site for broadcast or CG development. Lighting, reflections and more. The THETA is super portable, low-cost, and can do bracketing of multiple images that are then combined in Photoshop. Really cool to meet so many people using the THETA so heavily.

Posting just a few photos from SIGGRAPH:

Headed to the show, grabbing a quick snack and charging up my THETA in SFO:

Busy booth, TK Tone, THETA product manager, doing multiple demos:

I’m in Canada, so I have to have at least one meal at Tim Horton’s:

[codetricity]

@jcasman

Lots of VFX people who used the THETA “every day,” especially to quickly grab HDRi data for use on site for broadcast or CG development. Lighting, reflections and more. The THETA is super portable, low-cost, and can do bracketing of multiple images that are then combined in Photoshop. Really cool to meet so many people using the THETA so heavily.

This is great information. I understand about using the THETA to grab HDRI data.

What would they use the 4K 360° live streaming for?

Do they need the live streaming into the HTC Vive headset? Does it serve any type of demo or mockup purpose in the VFX industry?

[jcasman]

VFX guys like to capture HDR images with the THETA. They can light CG models using 360 HDRs. They get accurate lighting direction and intensity very quickly this way.

I believe they do not use live streaming at all.

[codetricity]

@jcasman

I believe they do not use live streaming at all.

Wow, that’s a bit discouraging. Was there interest in the HTC Vive demo at SIGGRAPH? If there was, what is the possible application? Maybe they can use the live streaming as a form of visual monitoring?

If there wasn’t interest in the HTC Vive demo, we can try and build more examples of HDRI use, which is what I’m trying to do now.

As far as HDRI, there appears to be platforms like Unity and a set of platforms like Maya and Blender. We have some Unity examples to get the lighting information.

I’m thinking of going through this Blender example:

[2.79] How to Use HDRI Background Images in Blender

[Tsikinator]

Hello, I tried to use both the tutorial and the executable app, but I encounter heavy lags. In unity, the fps that is displayed is ~17-19. I am using the provided cable to connect V with my laptop, which has the following specs: Intel Core i7 7700HQ @ 2.80GHz, 4095MB NVIDIA GeForce GTX 1080 (ASUStek Computer Inc), 16GB Ram.
I aim to use Theta V for a Telemetry project, so any help / insight would be useful

[codetricity]

To help us replicate the test, can you tell us what technique you’re using to measure the framerate?

[Tsikinator]

Besides getting the choppy feeling in the headset, I used the stats tab from Unity’s game menu and: https://assetstore.unity.com/packages/tools/integration/lite-fps-counter-probably-the-world-s-fastest-fps-counter-132638 , which I use in building most projects.

[codetricity]

On Windows 10, are you using the driver below (RICOH THETA V 4K), or are you using your own solution with the THETA V driver? You’re still getting the low framerate with (RICOH THETA V FullHD)? The FullHD one is 2K.

https://topics.theta360.com/en/faq/c_06_v/304_1/

[Tsikinator]

Thank you so much for your help, both here and in the DMs. I ended up using FullHD, with which I get 70-90 fps and no “delay” within the sphere. 4k still behaves as before, with low fps and black dragging spots when repositioning the head.

[Doscrash]

Hi, I purchased a Ricoh Theta V just because of this thread but I get the same choppiness that Taikinator experiences when playing the test binary. Do you know how to fix this? Is it using the wrong resolution?

[codetricity]

I need to test this again with the newest version of Unity. There may be some problem with the demo now. The maximum fps that you could expect is 30fps under ideal conditions. When you turn your head rapidly there is some lag and choppiness.

You can see from the post below what a research group in Tokyo is doing to transmit the stream over a WAN.

Telepresence Using THETA V and Segway by TwinCam Go

A community member pointed out this amazing project using two THETA V cameras. It won an award at Innovation 2018 by a governmental agency. The project was also nominated for SIGGRAPH Emerging Technologies next year. I believe the groups behind this project are: Tokyo Metropolitan University Ikei Research Laboratory NTT Communication Basic Science Laboratory [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] …

When the camera moves, they are reducing the resolution to 3K.

THETA Stereo Vision and Tactile Feedback Transmission Over The Internet with TwinCam Go

[image] We’re closer than you think. In the movies, VR transports people to a world filled with sights, sounds, and touch. As software developers and hardware engineers, we know that world is far away. Or, is it? theta360.guide interviewed Professor Yasushi Ikei of Tokyo Metropolitan University about TwinCam Go, his research project that transmits vision, sound, and motion over wide-area networks. Professor Ikei has an Engineering Ph.D. from Tokyo University and conducts research on virtual r…

Update 2/27/2019 I just tested this again with 4K video. Although it’s not as smooth as using something like VLC locally, it’s fairly smooth. I’m not getting any of the black dots.

Can you take a screencapture of the problem you’re facing?

---

Update March 4, 2019 I was at a Unity development meetup yesterday with @jcasman. The developer that presented mentioned that there were problems with different versions of SteamVR. He suggested using SteamVR from GitHub, not from the Steam Assets store. As the demo is old, it’s possible that the version of SteamVR it is using is old and needs to be updated.

In our tests, the base stations need to be set up for the demo to work. If the base stations lose connection with the headset, the demo will glitch.

There is lag. It’s not usable for an action game. The camera feed is more suitable for observation and visual documentation. It can be used for telepresence if latency is not a problem for your application. If you require precise timing, it is not a good solution.

[ZHAOYANBO123]

Hello! I want to use the RICOH R Dev Kit and HTC VIVE headset to implement VR in Unity.I followed your steps, but the live steam did not show in VIVE.