Telepresence Tips for HTC Vive?

codetricity · April 11, 2018, 12:44am

Question 1: Matching height of telepresence world with physical world

I’m using HTC Vive with the THETA V for telepresence. Is there a recommended height for the physical THETA V camera and the corresponding position of the SteamVR virtual camera rig inside of Unity?

Right now, I’m getting motion sickness because the virtual ground in the headset does not match up with the real ground under my feet. If the headset is at a physical height of 170cm, should I physically put the THETA V at 170cm on a tripod?

Question 2: Transmitting THETA V live video stream over network IP

I would like to display the THETA V connected to one Windows 10 computer to another Windows 10 computer that has a HTC Vive connected to it. Has anyone successfully done this? I’m thinking of testing a USB or IP solution.

These are the ones that I’ve found for possible testing:

Please advise if you’ve tested one of these or have had any success with any other solution.

In the setup I envision, I am planning to test two high-powered Windows 10 machines and GigE connected to a hub, so the hardware of the transmitting and receiving machines should be okay. I’m skeptical that I can push a 4K video stream over the network, but may try it.

xwindor · April 11, 2018, 6:20pm

This sounds like a really cool project! I don’t know much about coding but from what I have learned making my own VR video content a general rule is to always have the VR camera set up at eye height.

I have tried putting the tripod much lower and found that it makes me feel miniature, which could also be interesting if there is no motion sickness.

codetricity · April 11, 2018, 6:33pm

Thank you for this tip. I am going to borrow a tripod from @jcasman today and we’re going to try it with the THETA V at eye-level.

I can display the THETA V inside of the HTC Vive headset at 4K. It’s a good experience. This is a screenshot of the desktop application I am working on that mirrors the headset.

This is a view that I took when I used the controllers to take a screenshot from inside of the headset. Now that I look at it, the view looks tilted. Maybe I need to adjust the rotation of the virtual camera. I’m using a scale of 8 for the size of the sphere.

Once I figure out some good settings for the virtual camera, sphere size and physical THETA V, I’ll write up my findings and post an article.

I haven’t configured the virtual menus yet to control objects inside of the telepresence sphere yet. If anyone has a starter project, they want to share with good base settings for the THETA V, please share.

I think that I want the user of the HTC Vive to be standing up in a small space and not use room scale VR features such as walking right now. I do not have the THETA V on a robot right now, so I think there is no advantage to physically walking. Though, I suppose I could map it to the zoom of the stationary physical camera.

Any advice on telepresence would be appreciated. Thanks.

Update after initial tests

Setting camera rotation to 180, -135, 0 yields good results.

I’ve set the sphere rotation to 0, 0, -13.

The HTC Vive default position is pointed in the same direction as the rear THETA V camera lense. The rear lense is the one that is facing away from the shutter button.

theta-v-htc-v-direction

If a person stands in front of the rear lense of the camera, then the person wearing the headset will feel like they are standing in front of the person and talking to them.

jcasman · April 12, 2018, 4:41pm

@xwindor How’s the project coming? I spent an hour or so in HTC Vive yesterday, using the live streaming setup that @codetricity has built. It’s cool! Just adding a few observations on tripod and height:

Yes, probably setting the height level at eye level is the most comfortable. I think a lot of people have issues with VR googles making them either slightly uncomfortable or quite nauseated. Eye level helps, I believe.
One observation: With the RICOH THETA V, it feels like the point of view (POV) is slightly below the lens, maybe at the bottom of the camera. I do not have an explanation for this, but I noticed it.
We tested both eye-level and low to the ground. @codetricity was worried that low to the ground would feel strange. For me, I enjoyed the different perspective. Live streaming from eye level (within in the same room, also) seemed kind of normal. But live streaming from very low to the ground was like being turned into a small animal or something. Different perspective, pretty interesting.
Yes, the low perspective did partially make me feel like I was standing in a hole.

xwindor · April 12, 2018, 5:01pm

This is really cool stuff! Going to keep an eye on this to see what you guys manage. I’ve just started dabbling in Unity as a way to present my stereo-graphic time-lapse project I’ve been working on. Have fun guys, this website has been an incredible resource!

codetricity · April 12, 2018, 5:52pm

I did another test and it now seems to work with the Y axis rotation of the CameraRig set to zero. If this continues to work, i will be simpler to set up.

The orientation of the 3D audio is messed up right now. I just realized today that if I mess with the rotation of the video stream, I need to adjust the spatial audio as well.

For the current HTC Vive demo, I have the spatial audio turned off. However, I think it’s a useful thing to implement to show off the capabilities of the THETA V spatial audio.

rickyief · August 14, 2018, 3:21pm

Hi, have you ever tried the IP video streaming of any of the options you mentioned?

Thanks

codetricity · August 14, 2018, 4:14pm

I have not tried any of software packages above. I am working with the Wireless Streaming Plug-in now to stream to YouTube

I can also connect the THETA V directly to Ethernet with USB OTG and stream over Ethernet direct to YouTube.

For shorter distances of 100 feet, it’s possible that a USB to CAT5 solution might work.

If you get something working, please report back.

rickyief · August 16, 2018, 8:02am

Thanks Codetricity, but I was refering to stream the Theta V to the HTC Vive over the network, is this possible too?

codetricity · August 16, 2018, 3:53pm

Sorry, but I have not tried this yet.

Do you have specific latency requirements? Is the network connected to the same switch? Or, are you going out over the Internet?

Is it okay to have a delay of 8 seconds between the camera and headset? Or, do you need something under 1 second?

There’s various issues in reducing the latency over the WAN. A group at GITAI is working on solutions to the problems.

rickyief · August 18, 2018, 11:19am

I wanted to make a proof of concept for 5G mobile network, so I would like to show that real-time Telepresence can be achieve in 5G. I’m not very familiar with VR development and for what I have research VR streaming over network is not really mature. Your project is the closest thing I have found for what I intent to do, it’s just missing the network part. If you ever try those remote USB over network with your solution please post your results.

Thanks

codetricity · August 20, 2018, 5:31pm

I believe that the GITAI OS can do this, but I don’t think they released it for developers. See the note from their site on latency.

GITAI could minimize 2.7K resolution data of 360-degree camera from 800Mbps to 2.5Mbps and data latency to 100ms via standard WIFI networks.

If you want higher resolution and are OK with higher latency, you can probably use more standard technology. However, we don’t have any examples right now. If we find someone with something working, I’ll post a note here.

If the latency is not that big an issue, can you use YouTube or Facebook Live Event? You can use the Wireless Live Streaming Plug-in to stream directly from the camera to YouTube. You can then use YouTube VR inside of the HTC Vive headset.

If you want to bypass YouTube/Facebook and build your own implementation, I outlined some of the limitations and considerations from my perspective. It may save you some time in your own research.