HowTo: Viewing 360° Video in Real-Time from THETA V with Oculus Go Browser

oculus
vr
oculusgo
livestreaming

#1

Viewing 360° Video in Real-Time from THETA V with Oculus Go Browser

f6dde443617ce02b9c37bb46b2ccf3c9-66x66

Original article in Japanese by @massie_g (Masashi Ganeko), Infocom Corp.

Translation by @jcasman. Many thanks to Masashi for posting this information!


Introduction

Oculus Go’s browser is based on Chromium and supports WebGL. WebVR can be used by using WebVR library A-FRAME.

Source: “Summary of Oculus Go browser from a Web Developer’s Perspective” [JAPANESE]

Also, real-time communications using WebRTC are possible based on experiments by @gtk2k and others.

TWEET: If you have Oculus Go, I’d like to check if the browser of Oculus Go supports WebRTC and getUserMedia, so please access the following address from Oculus Go’s browser. https://turbographics2000.github.io/webrtc_simplecheck/ If everything’s green, then it’s OK.

gtk2k%20tweet1

TWEET: It looks like this. Once you tap it, the microphone’s permission dialog will appear, and if you allow it, the loopback also comes through.

yoshitaka%20tweet1

RETWEET: Oculus Go WebRTC support works!
TWEET: Success! I can voice chat between PC and Oculus Go.

gtk2k%20tweet2

Therefore, if you use a RICOH THETA V which can be recognized as a web camera via a USB connection, you should be able to see real time 360 degree video with the Oculus Go browser.

Try Moving Samples

I made samples in order to experience THETA V real-time 360 degree video with Oculus Go. It is published on GitHub Pages.

Broadcasting from a PC

  • Connect RICOH THETA V to the PC via USB, turn on the power and set it to Live mode

  • Open https://mganeko.github.io/aframe/pc.html with Chrome browser on your PC

    • The room is randomly allocated. This is possible to change.

    • By attaching ?apikey=ZZZZZZZZZ to the URL, you can use your SkyWay API KEY (Please allow it to use in the mganeko.github.io domain beforehand)

Skyway1

  • Click the [Get Devices] button to get a list of available Video devices (camera) and Audio devices (microphone)

    • Please be advised that you will be asked for access to the camera and microphone

    • Once THETA V is found, it will be selected automatically

    • If it is not detected even when the THETA V is connected, please exit Chrome and reboot again

  • Please click the “Start Video” button

    • Video and audio are acquired and displayed in the browser
  • Please click the “Connect” button

    • Connect to SkyWay and join the designated Room

    • The URL for connecting with Oculus Go is displayed below your video

    • If you open this URL in Oculus Go browser (or another WebVR compatible browser) you can watch the video

  • To stop broadcasting, click [Disconnect] → [Stop Video] in that order

Viewing through Oculus Go

  • Please start up your Oculus Go browser (or another WebVR compatible browser)

  • The URL displayed on the broadcast side is accessed by the browser

  • It will take a little while to load

    • When A-FRAME is ready, a message will be displayed in the center (Please click the window to start 360 video streaming)
  • Please click the screen (pull the trigger on the Oculus Go controller)

    • Connect to SkyWay and join the designated Room

    • The image from the THETA V 360 connected to the PC is displayed

  • Clicking the glasses type VR mode button on the bottom right changes it to 360 degree mode (align the cursor and pull the trigger on the Oculus Go controller)

    • Depending on the direction your head is facing, the viewing direction of the image changes
  • To exit 360 degree mode, press the [Back] button on the controller

  • The process of disconnecting from the Skyway connection is not completed yet. Please reload or move to another page

Displaying 360° Videos using A-FRAME

A-FRAME is a framework that lowers the barriers to WebVR. Just even using Oculus Go’s browser, you can see A-FRAME samples in 360 degree 3D. Some of the samples even show saved 360° videos.

Also, even though it is not Oculus Go, there are attempts at displaying RICOH R’s videos on WebVR using A-FRAME.

Based on this article, it was possible to see 360 degree real time videos in WebVR. It seems possible to do this.

On 360 Degree Video Formats

Roughly divided, there are two types of 360 degree images

  • Dual fisheye Format (2 fisheye images) - This is the format when connecting by USB to THETA S

  • Equirectangular Format (panorama) - This is the format when connecting by USB to THETA V, RICOH R

Since A-FRAME is compatible with Equirectangular Format, THETA S cannot be used as-is. Although it is unconfirmed, if RICOH THETA UVC Blender is used, it seems that output from THETA S could be converted into equirectangular format and used.

Display of 360 Degree Video using A-FRAME

Exact details will be left to another A-FRAME focused article, but in order to display 360 degree video, the following code can be used.

<a-scene>
    <a-assets>
      <video id="video1" autoplay loop crossorigin="anonymous" playsinline webkit-playsinline></video>
    </a-assets>
    <a-videosphere src="#video1" rotation="0 180 0">
    </a-videosphere>
</a-scene>

Write the original video tag in the a-assets tag and reference the id of the video tag from the a-videosphere component. Normally we would specify the URL in video.src, but if we use WebRTC we will pass MediaStream to video.srcObject.

Version of A-FRAME

The latest version of A-FRAME is 0.8.0 (as of May 2018). However, as much as I tested with this version 360 degrees video could not be displayed well. I tried 0.7.1 and because I can display without a problem, I am using that.

About Using Something other than Oculus Go to View

A-FRAME can be used with other devices as long as it has a browser that supports WebVR.

  • Chrome or Firefox on PC (Windows / Mac / Linux)

    • Although it is unconfirmed, you should be able to see 360 degree VR on Oculus Rift or HTC Vive
  • Chrome/Firefox on Android

    • You can see 360 degree VR using Google Cardboard
  • Chrome/Firefox/Safari on iOS

(Since I changed the model to iPhone 6, the characters were small and it was hard to see, so I used the magnified display mode, which was actually the cause. For those who suffer from the same problem, I will describe how to change the setting.)

No, it’s not close to the convenience of Oculus Go, but it can be done with other devices.

Video / Voice Communication Using WebRTC

When it comes to real-time video / voice communication with a web browser, WebRTC is the answer. Signaling servers and STUN / TURN servers are required. Although it can be setup by oneself, this time we used a service offered by NTT Communications, prioritizing actual speeds.

There are quite a few articles about using WebRTC. It’s a little bit old, but I wrote up some information on this, so please have a look.

Acquisition and Selection of Video / Audio devices

Although not mentioned in the WebRTC Introduction 2016 link up above, you can detect the video device (camera) and audio device (microphone) connected to the PC.

 function getDevice() {
    // Acquire video and audio to obtain the name of camera and microphone (ask user for permission)

    // -- enum device --
    navigator.mediaDevices.getUserMedia({video: true, audio: true})
    .then(stream => callBackDeviceList(stream))
    .catch(err => console.error(err));
  }

  // When you receive permission from the user, obtain a list of devices
  
 function callBackDeviceList(stream) {
    navigator.mediaDevices.enumerateDevices()
    .then(function(devices) {
      devices.forEach(function(device) {
        if (device.kind === 'videoinput') {
          const id = device.deviceId;
          const label = device.label || 'camera' + '(' + id + ')';
          // .... Processing when video device is detected
        }
        else if (device.kind === 'audioinput') {
          const id = device.deviceId;
          const label = device.label || 'microphone' + '(' + id + ')';
          // .... Processing when audio device is detected
        }
      });
    })
    .catch(function(err) {
      console.error("getDevices ERROR:", err);
    });

    stopLocalStream(stream);
  }

  // Stop video and audio
  function stopLocalStream(stream) {
    let tracks = stream.getTracks();
    for (let track of tracks) {
      track.stop();
    }
  }

To specify the Video / Audio device, use the detected id.

  • Audio device id: xxxxxxx

  • Video device id: yyyyyyy

    • Desired width, height: wwww, hhhh

With this information, specify as follows:

let options = {
  audio : {
    deviceId : {exact: "xxxxxx" }
  },
  video : {
    deviceId : {exact: "yyyyyy" },
    width : { min : wwww, max : wwww },
    height : { min : hhhh, max : hhhh }
  }
}

navigator.mediaDevices.getUserMedia(options)
.then( stream => {
  // ... Successful acquisition of media stream

})
.catch( err => {
  // ... When there’s an error
});

Conclusion

Unity and other applications are required to develop Oculus Go applications, but WebVR can be developed using just a text editor. Real time communication is also possible with WebRTC. If you use A-FRAME, you can also use devices other than Oculus Go. Please try it out!


Ricoh Theta S live streaming to htc Vive for drone
#2

Great stuff. As this appears to push the stream to Skyway servers, is there an advantage to stream the video using Skyway versus YouTube or Facebook Live 360 Events.

I would expect the Oculus Go to work with Facebook Live 360 Events due to the connection between Oculus and Facebook. However, I have not tested it.

I’m curious about:

  1. latency
  2. resolution
  3. noise or graphic artifacts