Solved: Unity Can't Display THETA V Live Stream on Windows 10

@Litto
Create the script in C# (or language of choice) and then attach it to the virtual sphere. You can drag and drop it.

Here’s a basic script that will recognize the camera. I have not implemented audio yet per the @Nima suggestion. I will try it today. If you successfully add audio to the script below, post the full working example. Thanks.

using UnityEngine;
using System.Collections;

public class webCamDetect : MonoBehaviour
{
	string camName;
	public const string RICOH_DRIVER_NAME = "RICOH THETA V 4K";
	// change to "RICOH THETA V FullHD" for lower resolution 
	// (and thus smaller data size)

	void Start()
	{
		WebCamDevice[] devices = WebCamTexture.devices;
		Debug.Log("Number of web cams connected: " + devices.Length);
		for (int i = 0; i < devices.Length; i++)
		{
			Debug.Log(i + " " + devices[i].name);
			if (devices[i].name ==  RICOH_DRIVER_NAME)
			{
				camName = devices[i].name;
			}
		}

		Debug.Log("I am using the webcam named " + camName);

		if (camName != RICOH_DRIVER_NAME)
		{
			Debug.Log("ERROR: " + RICOH_DRIVER_NAME +  
				" not found. Install Ricoh UVC driver 1.0.1 or higher. Make sure your camera is in live streaming mode");
		}

		Renderer rend = this.GetComponentInChildren<Renderer>();
		WebCamTexture mycam = new WebCamTexture();

		mycam.deviceName = camName;
		rend.material.mainTexture = mycam;

		mycam.Play();
	}
}

Is the spatial audio input from the THETA V working with the HTC Vive or Oculus Rift (or other) headset? Is it 4 channel audio or 2 channel?

I tried your script - I get an error
Assets/testScriptBasic.cs(40,10): error CS1525: Unexpected symbol `end-of-file’

I used this script, which worked fine (without audio):

using UnityEngine;
using System.Collections;

public class testScriptBasic : MonoBehaviour {

void Start() {
    WebCamDevice[] devices = WebCamTexture.devices;
    Debug.Log("Number of web cams connected: " + devices.Length);
    Renderer rend = this.GetComponentInChildren<Renderer>();

    WebCamTexture mycam = new WebCamTexture();
    string camName = devices[2].name;
    Debug.Log("The webcam name is " + camName);
    mycam.deviceName = camName;
    rend.material.mainTexture = mycam;

    mycam.Play();
    }
}

Update on Spatial Audio Tests

to followup on the discussion between @nima and @Litto

  • I have the THETA working as a spatial audio microphone inside of Unity
  • I can play spatial audio on headphones attached to my computer
  • I can get sound inside of the HTC Vive headset
  • I have not done precise tests of the orientation of the spatial sound inside the headset

Script for THETA V Live Streaming plus basic audio

using UnityEngine;
using System.Collections;

public class webCamDetect : MonoBehaviour
{
	string camName;
	public const string RICOH_DRIVER_NAME = "RICOH THETA V 4K";
	// change to "RICOH THETA V FullHD" for lower resolution 
	// (and thus smaller data size)

	// Audio
	public const int THETA_V_AUDIO_NUMBER = 0;   
	AudioSource audioSource;

	void Start()
	{
		WebCamDevice[] devices = WebCamTexture.devices;
		Debug.Log("Number of web cams connected: " + devices.Length);
		for (int i = 0; i < devices.Length; i++)
		{
			Debug.Log(i + " " + devices[i].name);
			if (devices[i].name == RICOH_DRIVER_NAME)
			{
				camName = devices[i].name;
			}
		}

		Debug.Log("I am using the webcam named " + camName);

		if (camName != RICOH_DRIVER_NAME)
		{
			Debug.Log("ERROR: " + RICOH_DRIVER_NAME +
				" not found. Install Ricoh UVC driver 1.0.1 or higher. Make sure your camera is in live streaming mode");
		}

		Renderer rend = this.GetComponentInChildren<Renderer>();
		WebCamTexture mycam = new WebCamTexture();

		mycam.deviceName = camName;
		rend.material.mainTexture = mycam;

		mycam.Play();

		// audio
		// this section working with HTC Vive, but have not 
		// verified spatial audio. Maybe try STEAM AUDIO?
		// https://valvesoftware.github.io/steam-audio/downloads.html
		// It's good enough for telepresence demo right now, but
		// I would like to tune the spatial audio
		//

		audioSource = GetComponent<AudioSource>();
		string[] audioDevices = Microphone.devices;

		for (int i = 0; i < audioDevices.Length; i++)
		{
			Debug.Log(i + " " + audioDevices[i]);
		}
		audioSource.clip = Microphone.Start(audioDevices[THETA_V_AUDIO_NUMBER], true, 10, 44100);
		audioSource.loop = true;
		while (!(Microphone.GetPosition(null) > 0 )) { }
		audioSource.Play();
	}
}

Using Default Audio Settings

image

I think I might have forgot the last curly bracket. I just posted a new script with the audio working.

This is what I have when outputting to HTC Vive

image

My recording devices
image

My script debug log from Unity

image

Note that the script is finding the THETA V first. Thus, in the script, it is array element zero and the HTC Vive microphone is array element one.

I have an Oculus Rift
so 2 Mics see screenshot.

Problem is. Performance starts lagging (image) and there is still no audio :confused:
See Screenshot!) I did everything the way you did.
I also used “RICOH THETA V FullHD” in the script. still lagging…
Maybe you know what is missing… :slight_smile: thx a lot!

On your machine, you have the driver or a physical Kinect for Windows USB audio.

Thus, you need to change the array index to 1, not 0.

In the code, change this to “1”

// Audio
public const int THETA_V_AUDIO_NUMBER = 0;   

On Windows 10, right-click on your speaker.

Verify the Playback and Recording devices.

Do you have “RICOH THETA V” as the microphone and Oculus Rift as the Playback device?

When you say, “lagging,” do you mean the video is not smooth?

Or, do you mean that there is latency between the time the real-world person moves and you see the movement in the virtual world?

Hello, here is my full script:

public class testScriptBasic : MonoBehaviour {

void Start () {
	WebCamDevice[] devices = WebCamTexture.devices;
	Debug.Log("Number of web cams connected: " + devices.Length);
	Renderer rend = this.GetComponentInChildren<Renderer>();

	WebCamTexture mycam = new WebCamTexture();
	string camName = devices[1].name;
	Debug.Log("The webcam name is " + camName);
	mycam.deviceName = camName;
	rend.material.mainTexture = mycam;
    mycam.Play();

    foreach (string device in Microphone.devices)
    {
        Debug.Log("Name: " + device);
    }
        var audio = GetComponent<AudioSource>();
   
    audio.clip = Microphone.Start("2- RICOH THETA V", true, 10, 44100);
    audio.loop = true;
    while (!(Microphone.GetPosition(null) > 0)) { }
    audio.Play();
    Debug.Log("number " + audio.clip.channels);

         
}    	

}

2 Likes

I am using both the HTC Vive and the Oculus Rift but I got only a 1 channel audio on live streaming. I haven’t been able to access to live spatial audio and I am not sure the THETA V allows us to do it.

N

I will try and let you know! Thank’s!

1 Like

Love it. Thanks so much for sharing this.

When you start your application, does SteamVR automatically configure your audio input as the THETA V? I’m reading this thread on Reddit and it seems like the settings in the screen below may not be saved.

I don’t have the headset plugged in now.

image

You’re right about the single channel audio for live streaming right now. I just remembered that I wrote this post back in Oct 2017.

I’m going to ask my friend at Ricoh about this. I think the camera can support this feature, but the internal firmware needs to be improved.

1 Like

OK I managed to have audio. However, the sound has a latency of 20 sec or so.
The lagging part of the video was inside unity, but in the built application it does work smooth, not perfect but ok.
Is there any change I can stream sound and image simultaneously?
We getting somewhere interesting, I feel like.

Nice greetings from Vienna :wink:

2 Likes

I’m not getting a 20 second lag between the THETA V and the HTC Vive, but there is some lag. I don’t have an Oculus Rift right now.

Out of curiosity, have you ever tried to get spatial audio into the headset from an external microphone?

As a test, when you run the THETA V into your headset, what does your CPU load look like? You can go into “Resource Monitor” on Windows 10.

In the picture below, my CPU load is under 10% when it processes the stream in OBS

This is on an old, mid-range CPU:

image

Using the THETA UVC 4K driver, your CPU will need to do some work. While, your application is probably not hitting the CPU limit, it’ll be good to rule out this possibility.

1 Like

One colleague of mine told me i should try changing the audio DSP Buffer to Best latency. Could not test it so far but I will let you know image

1 Like

Hello,

I did not need to change anything in the audio configuration of SteamVR, so I believe it is automatic.

N

1 Like

Thanks. I have it working with single channel audio.

I did a video review of a new Acer Predator Helios 300 gaming laptop with quad core i7. The CPU was maxed out at 100%.

I’m getting an audio delay of less than one second. I had the DSP Buffer Size set to default. I will try it with “Best latency”

If other people want to try it, the setting is in Edit → Project Settings → Audio → DSP Buffer Size

Even though the CPU was maxed out, the movement in the headset was smooth.

BTW, I think you’re aware, but the latency is a problem that for telepresence that many people are working on. One solution is to develop a new protocol with better compression, specifically for transmission over networks.

See this company for cool ideas:

http://gitai.tech/

Hello? Are you all still here?
I have a couple of questions. I’ve got 360 camera but it’s not THETA, it’s Insta360. So um… I can’t realize how you remove this black gaps on your sphere. What does this shader do?
(Solved: Unity Can’t Display THETA V Live Stream on Windows 10)
And what do you do in regedit? Is this thing for searching webcam or for… turning my webcam with 2 lenses into 2 webcams with single lenses?
I can’t resolve my problem with black gaps for a week.
Please help. Thanks in advance.

The THETA V outputs 4K equirectangular live streaming video. I’m not familiar with the insta360. You may want to ask someone on the insta360 developer site. If you do decide to purchase a THETA V, it also has the ability to develop plug-ins inside the camera.

Good luck with your project.

@codetricity Does THETA V support Linux ? Is there any possiblity that we can use your unityPack on Ubuntu?

Thanks In Advance