Solved: Unity Can't Display THETA V Live Stream on Windows 10

[craig]

@Christopher_Paton, thanks for reporting back on the success. It helps others.

The original Unity examples in this topic are old and I should reorganize the examples to help people find the updated information.

Let us know if you create something cool.

[Christopher_Paton]

Trying to set up telepresence robot operation.

Just trying to learn unity to get oculus quest controllers mapped.

[craig]

You may find these articles interesting.

Telepresence Using THETA V and Segway by TwinCam Go

A community member pointed out this amazing project using two THETA V cameras. It won an award at Innovation 2018 by a governmental agency. The project was also nominated for SIGGRAPH Emerging Technologies next year. I believe the groups behind this project are: Tokyo Metropolitan University Ikei Research Laboratory NTT Communication Basic Science Laboratory [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] [image] …

RICOH THETA Powers Construction Equipment VR Technology at Kumagai

[image] Just saw a RICOH THETA Z1 featured in a Japanese article about unmanned construction technology developed by Kumagai Group and Tokyo National College of Technology. The system provides audio and live streaming video with the RICOH THETA Z1 to remote operators. [image] The group developed a VR Cockpit to control construction equipment remotely using a telepresence experience. [image] Real-time video from the RICOH THETA Z1 is streamed into a VR head-mounted display or cockpit. Au…

---

The examples below are not using Unity. The developers for the two projects below are on this forum. You may be able to exchange ideas on implementation.

Successful Theta V stream from drone to VR headset 0.25 miles away

Just figured I would drop a little project update here letting people know about the senior design project I worked on that used a Ricoh Theta V. I know @codetricity and others on here would definitely be interested in it, since I have seen people talking about sending Ricoh Theta V video from a drone, but as far as I know noone has demonstrated anything yet. We built a system that would allow a ground operator to view a livestream video from a drone in their VR headset. The system will send th…

Industrial Robot Uses RICOH THETA 360° Live Video To Inspect Tight Places

[image] Hugues Perret builds robots that explore the inside of dark pipes under the streets of France. His robots live stream 360° video direct to a laptop, enabling the robot operator to see any cracks or damage. They can look up, down, left, right at any point in the inspection. His company, FOX SEWER ROVER, designs and builds these amazing robotic rovers for a leading French company that repairs a complex network of underground pipes. In addition to live streaming 360° video, the rover can…

[Christopher_Paton]

You sir, are a legend. thank you.

Chris

[anokhee]

Hello everyone.

I am using Theta V for a project, wherein I want to get the live stream from the camera on the screen of a phone, such that it can be viewed in real time using a VR headset (so the picture format has to be such). Could anyone please help me out?

Wired or wireless, either type of communication works. The only requirement is that the lags must be minimized. I am looking for some existing apps or methods which could help me achieve the above stated aims.

Thank you.

[craig]

You can test out the Lockheed Martin Amelia Drone app. View it in a browser and use A-Frame (included in the app) to set up the interface with the VR headset.

Otherwise, use the RTSP streaming method by @Hugues.

You can either run A-Frame in a browser or using a WebView (untested with the app) in your mobile project.

[anokhee]

Thank you.

The Amelia Viewer app did work for getting the live stream, with very less latency. However, we did not understand how A-Frame is used. I would appreciate if you could help us out with some guide or documentation to set up the interface with the VR headset/mobile phone.

I also wanted to ask how the VR headset/mobile phone is connected to the Amelia Viewer app. Is it a wired connection (through a USB cable) or is there some other way to connect them?

[craig]

There is extensive information here:

https://wiki.rit.edu/display/P203190/Amelia+Drone

RIT Amelia Drone Project (P19123)

[anokhee]

Thank you. The documentation was quite useful to understand how the interface with the VR headset is set.

However, we do not have an Oculus/Vive headset. So we are trying to look for a method which could help us do the same thing with any VR headset.

I would like to describe two tests that we conducted for the same.

Test 1: We tried the application that you had created for the project.

• We were able to connect and view the livestream from Theta V on the PC.
• We used Trinus VR software to view the livestream from the PC on the mobile device.
• However, the head tracking didn’t work.
• As we moved the phone around, the cursor on the PC moved, but the image didn’t. It is only when the cursor was clicked and dragged around, we could change the view of the image.
• I wanted to ask if there a workaround for that. We would want the view of the image to change as the phone is moved around.

Test 2: We tried to connect the phone by using the local network of the PC as a server. The source repository can be found here.

• The motion sensors could be accessed only over a secure connection, so we used a proxy server using https://

(The steps that we followed are enclosed here)

1. Download the source repo from here.

2. Open cmd.exe

3. Run: cd Downloads/amelia_viewer-master (Assuming the folder is in the Downloads folder)

4. Run: npm i

5. Run: npm start

6. ‘Ready for changes’ is displayed

7. The app opens up

8. Open another cmd.exe

9. Run: npm i -g https-proxy

10. Run: https-proxy --target=http://localhost:8181 --port=8182

11. ’ Proxing to “http://localhost:8181” through “htpps://localhost:8182” ’ is displayed

12. Open Firefox on the PC and go to http://127.0.0.1:8181/

13. Open the mobile browser

14. Open https://a.b.c.d:8182 (Replace it with your IP address)

• Head tracking worked perfectly here.
• However, when Theta V is connected to the PC, or when a local video is played on the PC, we could not view it on the mobile device.
• We could only see the first still image(the example) on the phone.
• We think that there is some part of the code that points to the example image and doesn’t change as the video is updated. However, we cannot understand what is causing this issue exactly.

There was an issue with the local video playback, which has also been attempted to be fixed in the source code that I have attached. But the loading timeout is 3000ms, so large videos could not be loaded. Increasing the timeout value also didn’t work for loading large videos. However, this is not a concern for us at the moment. I mentioned it because we were not sure if it is related to what is causing the issue with the livestream.

Could you please help us out here?

Thank you. Have a great day!

[Rob_Erta]

Hi everyone,
its 2020 and I feel like going back in time while not being able to get rid of the “Could not connect pins” error in unity.

I use Windows 10 with Unity 2020.1. I have updated the Ricoh Theta V to the latest firmware(3.50.1) and installed the current UVC blender driver.

Now I successfully managed to livestream to YouTube via OBS and also Unity recognizes the Theta.
However using the script attached to the sphere (as above and according to the unity documentation still up-to-date):

    Renderer renderer = this.GetComponentInChildren<Renderer>();
    WebCamTexture mycam = new WebCamTexture("RICOH THETA V/Z1 4K");
    renderer.material.mainTexture = mycam;
    mycam.Play();

gives me the error this whole thread started with:
“Could not connect pins - RenderStream()”

Anyone recently experienced the same? Is there any updated workaround?

[craig]

Are you using the updated driver? Lower right-hand corner of table. It is below the table.

https://support.theta360.com/en/download/

[dimdallas]

Hello @craig!

I have gone through the whole thread but I 've been experiencing the same problem as @Rob_Erta. I have installed the current Live-streaming app for Windows (RICOH THETA UVC V/Z1, v2.0.0).
OBS live-streaming works fine. Can there be a workaround?

[craig]

Hi, thanks for posting your question. It’s great to see Unity development still going strong.

I need to reinstall Windows 10 due to a hard disk problem and then I’ll run this again.

Can you post what version of the camera firmware, Unity, and Windows 10 you are using so that we have baseline information that might help trace the problem?

[dimdallas]

I am using Theta V with the latest firmware (3.50.1), Windows 10 Home edition and Unity Version 2019.4.6f1

[dimdallas]

Also @craig,
Would it be possible to give an answer to this comment of mine on Wireless streaming?

New Wireless Streaming Guide Available

Hello @codetricity! Is it possible that this wireless streaming approach with Theta V could work with a custom RTMP (like NGINX or MonaServer) rather than Youtube or Facebook? It would be even better if I could combine Wireless Live Streaming Plug-in with the THETA RTSP Streaming Plug-in (because I want least possible delay). Or the THETA RTSP Streaming Plug-in works wirelessly by default? Thank you in advance, Dimitris

Sorry for asking so many things at once,
DImitris

[pstawicki]

Hi @dimdallas,
I just tested some setup I was working on:
Windows 10 (20H2, build 19042.746) with WSL2 (windows subsystem for linux) with Ubuntu 18.04 LTS.
I installed NGINX (with rtmp module) on the Ubuntu (running in WSL2), example config, and on that same Windows machine watched the stream with ffmpeg/ffplay:
ffplay -fflags nobuffer -i "rtmp://192.168.2.118/live/z1"
z1 -> stream key.

The ffmpeg/fflpay works also for the “THETA RTSP Streaming Plug-In”.

Since this thread is about Unity, one could theoretically use FFMPEG as an plugin (dll) in Unity and project/pipe ffmpeg/ffplay output on a texture.
Piotr

[dimdallas]

@pstawicki

The ffmpeg/fflpay works also for the “THETA RTSP Streaming Plug-In”.

Hi @pstawicki,
I just tried the THETA RTSP Streaming Plug-In and I successfully viewed the stream with ffplay -fflags nobuffer -i rtsp://192.168.XXX.XXX:8554/live?resolution=1920x960

But I search for a solution for FFMPEG projection in Unity. Final target is either projecting on a inside-out sphere, or even better projecting on Skybox. I can’t find something for this problem, but I will keep on searching. If you have anything in mind, it will be usefull.

[craig]

See this project by @Hugues

Low latency (0.4s) H.264 livestreaming from Theta V WiFi to html5 clients with RTSP plug-in, ffmpeg and Janus Gateway on raspberry Pi

Hello, After many tries on Theta V, I achieved to get a H.264 livestreaming on a Raspberry Pi for multiples html5 clients with a 0.4s latency. It uses almost all realtime protocols available and powerfull softwares just to bypass USB livestreaming witch is not available on linux ! How it works: Theta V with RTSP plug-in connected in client mode (should work in AP mode too, 5GHz WiFi for better bandwith). RTSP with Device WebAPI Plug-in also works. ffmpeg (running on a Raspberry Pi) gets the…

it’s possible that the Janus WebRTC server is what you’re looking for.

https://janus.conf.meetecho.com/

the solution by @pstawicki using the RTMP module for NGINX looks like a great solution too.

[Rob_Erta]

Hi,
sorry for the huge delay. I was quite frustrated and put off. So indeed I used the link as described.
To be sure its not a different problem, I checked and confirmed that the YouTube LiveStream via OBSStudio works. So in principle my ThetaV allows live streaming, but it remains stubborn when it comes to Unity, even after adding DevicePath to the registry.

This is how the registry looks like:

Also I recorded a video of my workflow using the Theta in Live Video mode, maybe I am missing something … https://youtu.be/HVgttcTZ3Xw

I use the latest firmware on theta: 3.15.1
And Windows 10, 10.0.19042

[craig]

Uninstall the driver you are currently using and install the older driver. See the link below. We tested it with a V as well.

Can RICOH THETA Z1 be used in Unity as a webcam?

Hi everyone, We want to build a small VR conferencing app using Unity. I see that there were a lot of posts about connecting Ricoh Theta V with Unity via USB cable (like a webcam) to live stream 4K, but I couldn’t find any post about connecting the newer Z1 camera to Unity. Did someone successfully managed to connect it to Unity via USB? I like the larger sensor on Z1 and the image quality it produces, that is why I would like to go with it. Thank you and have a nice day.

RICOH THETA Z1 and V with Unity in 2021 for 360 Video Streaming