I think I found a pretty cool blog post on the topic of displaying live video streaming in VR goggles. It’s got some good tips and code samples.
There are a couple of big caveats. First, it’s not a nicely built app for a mobile device that you can easily display in Google Cardboard. Second, it’s Oculus, so I’m sure that’s still pretty hard for most people to get their hands on. And, third, though maybe less important, the blog post itself is in Japanese. But luckily I lived and worked in Japan for quite a few years and can translate. (Plus it’s a fairly technical blog post, so it’s more about code samples than translation nuances.)
I just found this, so I’ll do a super quick translation now, but I’ll try and come back to it soon.
It was originally posted to the Infocom Tech Blog, by Ganeko from Infocom.
The core idea is this image:
You can display live 360° video using WebRTC through Firefox with the WebVR plugin displayed in Oculus Rift DK2.
Oculus Prep
The blog skips over the details of setting up your Oculus. It mentions that getting all the right drivers for the graphics card installed and installing runtime “appears to take some effort.”
Browser Prep
For the browser, it used to be that Firefox Nightly was required, but since Firefox 42 it now handles WebVR. You need to install the Mozilla WebVR Enabler.
https://addons.mozilla.org/ja/firefox/addon/mozilla-webvr-enabler/
JavaScript Libraries
For JavaScript, you can go to the Three.js site to get the following:
For display effects: http://threejs.org/examples/js/effects/VREffect.js
For head tracking: http://threejs.org/examples/js/controls/VRControls.js
And from the Mozilla github repository: https://github.com/MozVR/vr-web-examples/tree/master/threejs-vr-boilerplate/js
(This is updated regularly so your code will probably stop working periodically so you will need to update your js files.)
JavaScript Code
You can find more information on WebVR environment setup here: https://developer.mozilla.org/en-US/docs/Web/API/WebVR_API/WebVR_environment_setup and here: https://webvr.info/
Please note: some of the // comments were in Japanese, and I’ve done my best to translate them, but I maybe be slightly off until I double-check.
First, you read in VREffect.js and VRControl.js
var vrEffect, effect;
var vrControls, controls;
var isVrMode = false;
// Adding initialization processing
vrEffect = new THREE.VREffect( renderer );
effect = renderer;
vrControls = new THREE.VRControls( camera );
controls = vrControls;
// Setting VR mode to on/off based on key events
function onkey( event ) {
if(event.keyCode == '79' || event.keyCode =='86') { // v || o
if(isVrMode) {
escapeVr();
isVrMode = false;
} else {
enterVr();
isVrMode = true;
}
}
event.stopPropagation();
}
window.addEventListener("keydown", onkey, true);
function enterVr() {
// check to see if we are in iframe before setting fullscreen.
if(!window.frameElement) {
controls = vrControls;
effect = vrEffect;
effect.setFullScreen( true );
}
}
function escapeVr() {
// check to see if we are in iframe before setting fullscreen.
if(!window.frameElement) {
//effect.setFullScreen( false );
effect = renderer;
if (document.webkitCancelFullScreen) {
document.webkitCancelFullScreen(); //Chrome15+, Safari5.1+, Opera15+
} else if (document.mozCancelFullScreen) {
document.mozCancelFullScreen(); //FF10+
} else if (document.msExitFullscreen) {
document.msExitFullscreen(); //IE11+
} else if(document.cancelFullScreen) {
document.cancelFullScreen(); //Gecko:FullScreenAPI interface
} else if(document.exitFullscreen) {
document.exitFullscreen(); // HTML5 Fullscreen API interface
}
renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(window.innerWidth, window.innerHeight);
}
}
// managing windows size
function update() {
if(isVrMode) {
// handling head tracking
if (typeof controls.update == 'function') {
controls.update();
}
}
else {
// handling the results of mouse actions
lat = Math.max(-85, Math.min(85, lat));
phi = THREE.Math.degToRad(90 - lat);
theta = THREE.Math.degToRad(lon);
camera.target.x = Math.sin(phi) * Math.cos(theta);
camera.target.y = Math.cos(phi);
camera.target.z = Math.sin(phi) * Math.sin(theta);
camera.lookAt(camera.target);
}
// video to image
videoImageContext.drawImage(localVideo, 0, 0, videoImage.width, videoImage.height);
if (videoTexture) {
videoTexture.needsUpdate = true;
}
//renderer.render(scene, camera); // making existing handling invalid
effect.render(scene, camera); // doing rendering for VR
}
Using this, you’ll be able to view your VR content in the Oculus Rift DK2. So you’ve got THETA S + WebRTC + WebGL + WebVR + Oculus and you can enjoy live streaming video in your goggles broadcast from somewhere else!