Connecting RICOH THETA Z1 with Unity

[Manuel_Couto]

Hi there!

Apologies for the delayed response, as I was down with a cold last week.

Were you able to get it working?

[craig]

@Manuel_Couto did you see this post on the 5GHz channel? Connecting RICOH THETA Z1 with Unity - #32 by craig

[Manuel_Couto]

I saw it, and I’m planning to try it out today.

I’m not sure if I’ll be able to post the results today or maybe just tomorrow, but I’ll get back to you as soon as I can.

Thanks for the help!

[HunterGeitz]

Thank you, I successfully found the new IP address of my Ricoh Theta Z1 by setting my phone and the Z1 to client mode and was able to use this IP address to fix the URL for OBS. It streams successfully but with around 2 seconds of delay. Any advice on how to improve this delay?

[biviel]

1. What would be your maximum acceptable latency?

2. Did you use a low latency 5ghz WiFi router placed close to you? There can be significant differences also depending how far are your devices from the WiFi router. Both camera and PC with OBS…

[craig]

We got less than 1 second latency between Hungary and Palo Alto. How far is the video being transmitted?

Sub-Second Low Latency 4K Live Streaming in HDR with FlowTours

FlowTours 4K HDR Live Streaming Low-Latency Demo Laszlo Vargas, CEO of FlowTours, demoed new low-latency functionality with the RICOH THETA Z1 and a prototype of the next version of his HDR Wireless Live Streaming Plug-in. We used the 360 live stream for video conferencing between Hungary and Palo Alto, California. The one-way stream between Hungary and Palo Alto showed less than 1 second latency. We were easly able to use it for both audio and video streaming using the Z1 360 camera and Z1 …

[biviel]

Now that I was able to do more tests, I can confirm that there are lot of aspect. Latency depends on bandwidth/bitrate too, so when video bitrate is lower, latency was lower too. During test we did there bandwidth and quality was lower. When I do a full 4 resolution, good quality stream latency is closer to 2 second end to end.

Now when I measured this latency on a WiFi 5 protocol 5ghz local network, I measured 800-900ms latency using 4k resolution and h.265 encoding when streaming to OBS Studio as Media Source.

[craig]

@biviel

Now when I measured this latency on a WiFi 5 protocol 5ghz local network, I measured 800-900ms latency using 4k resolution and h.265 encoding when streaming to OBS Studio as Media Source.

Is this when the Z1 and the the computer running OBS Studio are in the same room? I wonder if the virtual cam of OBS adds additional latency when it gets to Unity? When I did the test with Unity, I did not check for latency.

[HunterGeitz]

1. I don’t necessarily have a maximum acceptable latency but less than a second would be preferred.

2. For now I’m using my laptop’s hotspot which is only few feet away from the Z1 at most. My laptop’s hotspot is the only source of WiFi I can use while at my college (Rowan University). The reason I can’t use my colleges WiFi is because I can’t connect the Z1 to it through client mode likely due to security reason’s with it being a college network, but if you have any possible ideas on how I could connect the Z1 to my college WiFi then it’d be appreciated.

[HunterGeitz]

For now I’m using my laptop’s hotspot which is only few feet away from the Z1 at most. My laptop’s hotspot is the only source of WiFi I can use while at my college (Rowan University). The reason I can’t use my colleges WiFi is because I can’t connect the Z1 to it through client mode likely due to security reason’s with it being a college network, but if you have any possible ideas on how I could connect the Z1 to my college WiFi then it’d be appreciated.

[biviel]

@craig

Is this when the Z1 and the the computer running OBS Studio are in the same room? I wonder if the virtual cam of OBS adds additional latency when it gets to Unity? When I did the test with Unity, I did not check for latency.

Yes, same room. In past, ~3 firmware upgrades back, before Ricoh optimized WiFi usage because of overheating, this latency was lower for me.

Now I’m back to test WebRTC again and to try to play in browser full 4k, as good quality as possible. Latency is about 300ms.

Here you can see my desktop. In a browser running a counter and in another browser in front of it, the live 4k preview of my Z1 streaming into it WebRTC:

This is not supported yet in my plugin because of the use cases I was looking to cover in past and lack of h.265 encoding when using webrtc.

.

[craig]

When you use OBS on the same laptop that you’re using as the hotspot, what is the CPU load shown on OBS? Is it showing something reasonable like under 10%?

I didn’t measure the latency between the Z1 and Unity when using RTSP plugin, but 2 seconds seems high. Just using RTSP plugin at 4K, I’m getting under 500ms inside the same room and not going to Unity, just Z1 to computer only.

I believe the @Manuel_Couto has it working from the RTSP plugin to OBS and then to Unity and seems to have it under 2 second latency, right?

[Manuel_Couto]

Hello again,

I promised an update yesterday about the wireless connection, but I’m still working on it.

For now, using WiFi and a 2.4GHz connection, I measured approximately 1.01 seconds latency for 1920x960 resolution and about 1.42 seconds for 3840x1920. (This is from Z1 to Unity)

→ 1920x960: 18.60 - 17.59 = 1.01s

→ 3840x1920: 17.72 - 16.30 = 1.42s

Since I intend to use this for VR teleoperation and require very low latency, I have been studying using other protocols and implementing 5G.

Today, I tried to implement 5G by connecting the Z1 with an Ethernet cable to the router and then using 5G to connect to the PC, but I’m still encountering some issues.

I will keep you informed when I get it working.

PS: @craig, regarding the problem I was having with transitioning from the 2.4GHz to 5GHz band connection, it turned out to be an issue with the router. Today, I tried it out with another router, and it worked fine with 5GHz.

Also, please note that the tests above were done yesterday with the old router that was having problems, so they might not be the most trustworthy source of information. As soon as I get new results, I will post them.

[biviel]

You used RTSP Plugin, right? H264 encoding, 2k and 4k and I assume equirectangular projection.

I’m wondering, is equirectangular needed at all? With my plugin I’m able to stream double fisheye directly and it generates lower heat. Using my plugin and fisheye and 24FPS it would work for hours in a 26-28°C environment, using SRT. Regarding latency, h.265 is important because it requires lower bandwidth, that also helps a bit with latency. Honestly I don’t know what is that RTSP plugin capable of.

[craig]

@biviel

Using my plugin and fisheye and 24FPS it would work for hours in a 26-28°C environment, using SRT.

Does he have access to your plugin?

I believe he’s using stream in a headset. He’ll need to get the frames onto a sphere at some point.

There is a long thread about using Unity to stitch the stream. However, the stitch is not as good as the stitch from the camera.

Tutorial: Live Ricoh Theta S Dual Fish Eye for SteamVR in Unity

I’ve been working on a system for robotic control using the HTC Vive for a few months now. Part of this system involved live streaming 360 feed into a virtual reality headset, which was easier said than done. There are a lot of instructions/tutorials/guides out there but nothing that completely does what I wanted to do, so I figured I would write this up for anybody who wanted to do something similar using SteamVR or other VR API’s in Unity. I’ll update this post when my solution is posted to gi…

@biviel do you know the latency improvement if the camera doesn’t have to stitch to equirectangular?

[Manuel_Couto]

I’m new to the world of cameras and protocol communication, so I’m still learning as I go.

Currently, I’m using the THETA RTSP Streaming plugin. I’ve tested it with both 2k and 4k resolutions. I’m not certain about the encoding it’s using, nor am I sure how to check or change it.

As for the projection type, based on @craig’s comment, I believe I’m using the standard equirectangular projection.

@craig

I believe he’s using stream in a headset. He’ll need to get the frames onto a sphere at some point.

Yes, I capture the live feed with OBS, convert it into a Virtual Camera, and then project it onto a Unity Sphere, which I can view with the HTC Vive headset.

[HunterGeitz]

When using OBS on the same laptop as the hotspot, the CPU load shown in OBS varies from less than 1% to around 6%, so yes it’s reasonable. At the moment we’re having another issue regarding using the camera with Unity. We can only enter play mode in Unity successfully when the Ricoh Theta Z1 is wired to the laptop and not connected wirelessly. When trying to use the wireless connection or trying to enter play mode without the camera connected at all, Unity crashes when trying to enter play mode. We (@Caleb_Amadoro) believe this is from the code we are currently using to attach the Ricoh Theta live video feed to a material on the surface of a sphere. Could you let us know what code you’re using for the sphere material or any other code that could potentially fix this issue?

[Manuel_Couto]

@craig @jcasman @biviel I’m back with some updates!

Over the past few weeks, I’ve been focused on live streaming from my Z1 to a Unity app while aiming for minimum latency. Currently, I’ve achieved an average latency of 544 ms.

Here’s the setup:

• A computer (PC1) connected via cable to the Z1. On this PC, OBS converts the video feed into a virtual camera. Subsequently, I access the virtual camera through a Unity app, that utilizes WebRTC to transmit to a second PC (PC2).
• PC2 runs the client-side version of the ChatApp.
• Both PCs are connected via Ethernet to 5G routers (PC1 is connected to R1, and PC2 is connected to R2).

For testing purposes, I compare the stopwatch time at four different points:

• Stopwatch: Real-time value
• OBS: Capturing latency
• PC1: Latency between OBS and Unity communication
• PC2: WebRTC latency

Below is an image of the setup used:

Additionally, here’s a screenshot with some test results and average values:

Upon analyzing the Excel sheet, it’s evident that the major latency occurs during the capturing process (Z1 to OBS and OBS to Unity). Surprisingly, the transmission latency is only 60 ms.

Moving forward, I plan to focus on reducing the capturing latency as much as possible.

I’d love to hear your thoughts on this!

[craig]

This is great information.

Is there any difference if you stream 2K instead of 4K? I was just wondering as the stitching time for each frame is consuming a significant portion of the time needed to get output from the camera over USB.

Can a Mac run Unity? You may be able to avoid the virtual camera with OBS if you use a Mac for a test. I have not tried.

[HunterGeitz]

@craig When using OBS on the same laptop as the hotspot, the CPU load shown in OBS varies from less than 1% to around 6%, so yes it’s reasonable. At the moment we’re having another issue regarding using the camera with Unity. We can only enter play mode in Unity successfully when the Ricoh Theta Z1 is wired to the laptop and not connected wirelessly. When trying to use the wireless connection or trying to enter play mode without the camera connected at all, Unity crashes when trying to enter play mode. We (@Caleb_Amadoro) believe this is from the code we are currently using to attach the Ricoh Theta live video feed to a material on the surface of a sphere. Could you let us know what code you’re using for the sphere material or any other code that could potentially fix this issue?