This article originally appeared on Qiita in Japanese by mechamogera
I tried to display dualfisheye movie (movie before conversion) in RICOH THETA S with Three.js. While getting help with the calculations, I worked through this myself, so there may be some strange issues. Please remember to take notes as you go along. (The calculations are hard.)
- Source code: THETAのDualfisheye動画をthree.jsで視聴するためのサンプル · GitHub
- Action sample: http://bl.ocks.org/mechamogera/5635338a1075baadba0b
I watched MP4 movies with the above code and could properly display the THETA live streaming image acquired by WebRTC.
Map THETA S’s USB live streaming to a spherical map with a browser.English Translation | Original Japanese
Until distributing the whole spherical image with THETA S (3) I appreciate it because articles such as pasted images were displayed.
Code Explanation
Overview
Referencing Completely Understandable WebGL Programming Even for Beginners, Taking the First Step with Three.js I implemented three.js.
The configuration is as follows.
- Three.min.js: Main processing of Three.js Files you have acquired
- OrbitControls.js: Files you have acquired for grabbing the mouse with Three.js
- Index.html: HTML created to watch Dualfisheye video
- Theta-view.js: Dualfisheye JS created to watch videos
- M20.mp4: Dualfisheye sample movie to display
I created a sphere, placed the camera inside, pasted and pasted a video texture with the video tag as the source.
Paste the Texture
In theta-view.js the following part sets the UV.
if (i < faceVertexUvs.length / 2) {
var correction = (x == 0 && z == 0) ? 1 : (Math.acos(y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
uvs[ j ].x = x * (404 / 1920) * correction + (447 / 1920);
uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080);
} else {
var correction = ( x == 0 && z == 0) ? 1 : (Math.acos(-y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
uvs[ j ].x = -1 * x * (404 / 1920) * correction + (1460 / 1920);
uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080);
}
Magic numbers of 404 and 447 correspond to the following sizes.
However, it is a rough value because it does not measure exactly.
Arrangement of Dualfisheye
The base code refers to this code in the answer to the following question.
Javascript - Mapping image onto a sphere in Three.js - Stack Overflow
faceVertexUvs[ face ][ j ].x = face.vertexNormals[ j ].x * 0.5 + 0.5;
faceVertexUvs[ face ][ j ].y = face.vertexNormals[ j ].y * 0.5 + 0.5;
However, if you keep the above code, the image will be distorted.
Therefore, we calculate and correct as follows.
Correction calculation
Sample
index.html
<html>
<head>
<meta charset="utf-8">
</head>
<body>
<video id="video" autoplay loop style="display:none">
<source src="m20.MP4" type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
</video>
<script src="three.min.js"></script>
<script src="OrbitControls.js"></script>
<script src="theta-view.js"></script>
<script>
window.addEventListener( 'DOMContentLoaded', theta_view('video'), false );
</script>
</body>
</html>
OrbitControls.js
/**
* @author qiao / https://github.com/qiao
* @author mrdoob / http://mrdoob.com
* @author alteredq / http://alteredqualia.com/
* @author WestLangley / http://github.com/WestLangley
* @author erich666 / http://erichaines.com
*/
/*global THREE, console */
// This set of controls performs orbiting, dollying (zooming), and panning. It maintains
// the "up" direction as +Y, unlike the TrackballControls. Touch on tablet and phones is
// supported.
//
// Orbit - left mouse / touch: one finger move
// Zoom - middle mouse, or mousewheel / touch: two finger spread or squish
// Pan - right mouse, or arrow keys / touch: three finter swipe
//
// This is a drop-in replacement for (most) TrackballControls used in examples.
// That is, include this js file and wherever you see:
// controls = new THREE.TrackballControls( camera );
// controls.target.z = 150;
// Simple substitute "OrbitControls" and the control should work as-is.
THREE.OrbitControls = function ( object, domElement ) {
this.object = object;
this.domElement = ( domElement !== undefined ) ? domElement : document;
// API
// Set to false to disable this control
this.enabled = true;
// "target" sets the location of focus, where the control orbits around
// and where it pans with respect to.
this.target = new THREE.Vector3();
// center is old, deprecated; use "target" instead
this.center = this.target;
// This option actually enables dollying in and out; left as "zoom" for
// backwards compatibility
this.noZoom = false;
this.zoomSpeed = 1.0;
// Limits to how far you can dolly in and out
this.minDistance = 0;
this.maxDistance = Infinity;
// Set to true to disable this control
this.noRotate = false;
this.rotateSpeed = 1.0;
// Set to true to disable this control
this.noPan = false;
this.keyPanSpeed = 7.0; // pixels moved per arrow key push
// Set to true to automatically rotate around the target
this.autoRotate = false;
this.autoRotateSpeed = 2.0; // 30 seconds per round when fps is 60
// How far you can orbit vertically, upper and lower limits.
// Range is 0 to Math.PI radians.
this.minPolarAngle = 0; // radians
this.maxPolarAngle = Math.PI; // radians
// Set to true to disable use of the keys
this.noKeys = false;
// The four arrow keys
this.keys = { LEFT: 37, UP: 38, RIGHT: 39, BOTTOM: 40 };
////////////
// internals
var scope = this;
var EPS = 0.000001;
var rotateStart = new THREE.Vector2();
var rotateEnd = new THREE.Vector2();
var rotateDelta = new THREE.Vector2();
var panStart = new THREE.Vector2();
var panEnd = new THREE.Vector2();
var panDelta = new THREE.Vector2();
var panOffset = new THREE.Vector3();
var offset = new THREE.Vector3();
var dollyStart = new THREE.Vector2();
var dollyEnd = new THREE.Vector2();
var dollyDelta = new THREE.Vector2();
var phiDelta = 0;
var thetaDelta = 0;
var scale = 1;
var pan = new THREE.Vector3();
var lastPosition = new THREE.Vector3();
var STATE = { NONE : -1, ROTATE : 0, DOLLY : 1, PAN : 2, TOUCH_ROTATE : 3, TOUCH_DOLLY : 4, TOUCH_PAN : 5 };
var state = STATE.NONE;
// for reset
this.target0 = this.target.clone();
this.position0 = this.object.position.clone();
// so camera.up is the orbit axis
var quat = new THREE.Quaternion().setFromUnitVectors( object.up, new THREE.Vector3( 0, 1, 0 ) );
var quatInverse = quat.clone().inverse();
// events
var changeEvent = { type: 'change' };
var startEvent = { type: 'start'};
var endEvent = { type: 'end'};
this.rotateLeft = function ( angle ) {
if ( angle === undefined ) {
angle = getAutoRotationAngle();
}
thetaDelta -= angle;
};
this.rotateUp = function ( angle ) {
if ( angle === undefined ) {
angle = getAutoRotationAngle();
}
phiDelta -= angle;
};
// pass in distance in world space to move left
this.panLeft = function ( distance ) {
var te = this.object.matrix.elements;
// get X column of matrix
panOffset.set( te[ 0 ], te[ 1 ], te[ 2 ] );
panOffset.multiplyScalar( - distance );
pan.add( panOffset );
};
// pass in distance in world space to move up
this.panUp = function ( distance ) {
var te = this.object.matrix.elements;
// get Y column of matrix
panOffset.set( te[ 4 ], te[ 5 ], te[ 6 ] );
panOffset.multiplyScalar( distance );
pan.add( panOffset );
};
// pass in x,y of change desired in pixel space,
// right and down are positive
this.pan = function ( deltaX, deltaY ) {
var element = scope.domElement === document ? scope.domElement.body : scope.domElement;
if ( scope.object.fov !== undefined ) {
// perspective
var position = scope.object.position;
var offset = position.clone().sub( scope.target );
var targetDistance = offset.length();
// half of the fov is center to top of screen
targetDistance *= Math.tan( ( scope.object.fov / 2 ) * Math.PI / 180.0 );
// we actually don't use screenWidth, since perspective camera is fixed to screen height
scope.panLeft( 2 * deltaX * targetDistance / element.clientHeight );
scope.panUp( 2 * deltaY * targetDistance / element.clientHeight );
} else if ( scope.object.top !== undefined ) {
// orthographic
scope.panLeft( deltaX * (scope.object.right - scope.object.left) / element.clientWidth );
scope.panUp( deltaY * (scope.object.top - scope.object.bottom) / element.clientHeight );
} else {
// camera neither orthographic or perspective
console.warn( 'WARNING: OrbitControls.js encountered an unknown camera type - pan disabled.' );
}
};
this.dollyIn = function ( dollyScale ) {
if ( dollyScale === undefined ) {
dollyScale = getZoomScale();
}
scale /= dollyScale;
};
this.dollyOut = function ( dollyScale ) {
if ( dollyScale === undefined ) {
dollyScale = getZoomScale();
}
scale *= dollyScale;
};
this.update = function () {
var position = this.object.position;
offset.copy( position ).sub( this.target );
// rotate offset to "y-axis-is-up" space
offset.applyQuaternion( quat );
// angle from z-axis around y-axis
var theta = Math.atan2( offset.x, offset.z );
// angle from y-axis
var phi = Math.atan2( Math.sqrt( offset.x * offset.x + offset.z * offset.z ), offset.y );
if ( this.autoRotate ) {
this.rotateLeft( getAutoRotationAngle() );
}
theta += thetaDelta;
phi += phiDelta;
// restrict phi to be between desired limits
phi = Math.max( this.minPolarAngle, Math.min( this.maxPolarAngle, phi ) );
// restrict phi to be betwee EPS and PI-EPS
phi = Math.max( EPS, Math.min( Math.PI - EPS, phi ) );
var radius = offset.length() * scale;
// restrict radius to be between desired limits
radius = Math.max( this.minDistance, Math.min( this.maxDistance, radius ) );
// move target to panned location
this.target.add( pan );
offset.x = radius * Math.sin( phi ) * Math.sin( theta );
offset.y = radius * Math.cos( phi );
offset.z = radius * Math.sin( phi ) * Math.cos( theta );
// rotate offset back to "camera-up-vector-is-up" space
offset.applyQuaternion( quatInverse );
position.copy( this.target ).add( offset );
this.object.lookAt( this.target );
thetaDelta = 0;
phiDelta = 0;
scale = 1;
pan.set( 0, 0, 0 );
if ( lastPosition.distanceToSquared( this.object.position ) > EPS ) {
this.dispatchEvent( changeEvent );
lastPosition.copy( this.object.position );
}
};
this.reset = function () {
state = STATE.NONE;
this.target.copy( this.target0 );
this.object.position.copy( this.position0 );
this.update();
};
function getAutoRotationAngle() {
return 2 * Math.PI / 60 / 60 * scope.autoRotateSpeed;
}
function getZoomScale() {
return Math.pow( 0.95, scope.zoomSpeed );
}
function onMouseDown( event ) {
if ( scope.enabled === false ) return;
event.preventDefault();
if ( event.button === 0 ) {
if ( scope.noRotate === true ) return;
state = STATE.ROTATE;
rotateStart.set( event.clientX, event.clientY );
} else if ( event.button === 1 ) {
if ( scope.noZoom === true ) return;
state = STATE.DOLLY;
dollyStart.set( event.clientX, event.clientY );
} else if ( event.button === 2 ) {
if ( scope.noPan === true ) return;
state = STATE.PAN;
panStart.set( event.clientX, event.clientY );
}
scope.domElement.addEventListener( 'mousemove', onMouseMove, false );
scope.domElement.addEventListener( 'mouseup', onMouseUp, false );
scope.dispatchEvent( startEvent );
}
function onMouseMove( event ) {
if ( scope.enabled === false ) return;
event.preventDefault();
var element = scope.domElement === document ? scope.domElement.body : scope.domElement;
if ( state === STATE.ROTATE ) {
if ( scope.noRotate === true ) return;
rotateEnd.set( event.clientX, event.clientY );
rotateDelta.subVectors( rotateEnd, rotateStart );
// rotating across whole screen goes 360 degrees around
scope.rotateLeft( 2 * Math.PI * rotateDelta.x / element.clientWidth * scope.rotateSpeed );
// rotating up and down along whole screen attempts to go 360, but limited to 180
scope.rotateUp( 2 * Math.PI * rotateDelta.y / element.clientHeight * scope.rotateSpeed );
rotateStart.copy( rotateEnd );
} else if ( state === STATE.DOLLY ) {
if ( scope.noZoom === true ) return;
dollyEnd.set( event.clientX, event.clientY );
dollyDelta.subVectors( dollyEnd, dollyStart );
if ( dollyDelta.y > 0 ) {
scope.dollyIn();
} else {
scope.dollyOut();
}
dollyStart.copy( dollyEnd );
} else if ( state === STATE.PAN ) {
if ( scope.noPan === true ) return;
panEnd.set( event.clientX, event.clientY );
panDelta.subVectors( panEnd, panStart );
scope.pan( panDelta.x, panDelta.y );
panStart.copy( panEnd );
}
scope.update();
}
function onMouseUp( /* event */ ) {
if ( scope.enabled === false ) return;
scope.domElement.removeEventListener( 'mousemove', onMouseMove, false );
scope.domElement.removeEventListener( 'mouseup', onMouseUp, false );
scope.dispatchEvent( endEvent );
state = STATE.NONE;
}
function onMouseWheel( event ) {
if ( scope.enabled === false || scope.noZoom === true ) return;
event.preventDefault();
event.stopPropagation();
var delta = 0;
if ( event.wheelDelta !== undefined ) { // WebKit / Opera / Explorer 9
delta = event.wheelDelta;
} else if ( event.detail !== undefined ) { // Firefox
delta = - event.detail;
}
if ( delta > 0 ) {
scope.dollyOut();
} else {
scope.dollyIn();
}
scope.update();
scope.dispatchEvent( startEvent );
scope.dispatchEvent( endEvent );
}
function onKeyDown( event ) {
if ( scope.enabled === false || scope.noKeys === true || scope.noPan === true ) return;
switch ( event.keyCode ) {
case scope.keys.UP:
scope.pan( 0, scope.keyPanSpeed );
scope.update();
break;
case scope.keys.BOTTOM:
scope.pan( 0, - scope.keyPanSpeed );
scope.update();
break;
case scope.keys.LEFT:
scope.pan( scope.keyPanSpeed, 0 );
scope.update();
break;
case scope.keys.RIGHT:
scope.pan( - scope.keyPanSpeed, 0 );
scope.update();
break;
}
}
function touchstart( event ) {
if ( scope.enabled === false ) return;
switch ( event.touches.length ) {
case 1: // one-fingered touch: rotate
if ( scope.noRotate === true ) return;
state = STATE.TOUCH_ROTATE;
rotateStart.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY );
break;
case 2: // two-fingered touch: dolly
if ( scope.noZoom === true ) return;
state = STATE.TOUCH_DOLLY;
var dx = event.touches[ 0 ].pageX - event.touches[ 1 ].pageX;
var dy = event.touches[ 0 ].pageY - event.touches[ 1 ].pageY;
var distance = Math.sqrt( dx * dx + dy * dy );
dollyStart.set( 0, distance );
break;
case 3: // three-fingered touch: pan
if ( scope.noPan === true ) return;
state = STATE.TOUCH_PAN;
panStart.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY );
break;
default:
state = STATE.NONE;
}
scope.dispatchEvent( startEvent );
}
function touchmove( event ) {
if ( scope.enabled === false ) return;
event.preventDefault();
event.stopPropagation();
var element = scope.domElement === document ? scope.domElement.body : scope.domElement;
switch ( event.touches.length ) {
case 1: // one-fingered touch: rotate
if ( scope.noRotate === true ) return;
if ( state !== STATE.TOUCH_ROTATE ) return;
rotateEnd.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY );
rotateDelta.subVectors( rotateEnd, rotateStart );
// rotating across whole screen goes 360 degrees around
scope.rotateLeft( 2 * Math.PI * rotateDelta.x / element.clientWidth * scope.rotateSpeed );
// rotating up and down along whole screen attempts to go 360, but limited to 180
scope.rotateUp( 2 * Math.PI * rotateDelta.y / element.clientHeight * scope.rotateSpeed );
rotateStart.copy( rotateEnd );
scope.update();
break;
case 2: // two-fingered touch: dolly
if ( scope.noZoom === true ) return;
if ( state !== STATE.TOUCH_DOLLY ) return;
var dx = event.touches[ 0 ].pageX - event.touches[ 1 ].pageX;
var dy = event.touches[ 0 ].pageY - event.touches[ 1 ].pageY;
var distance = Math.sqrt( dx * dx + dy * dy );
dollyEnd.set( 0, distance );
dollyDelta.subVectors( dollyEnd, dollyStart );
if ( dollyDelta.y > 0 ) {
scope.dollyOut();
} else {
scope.dollyIn();
}
dollyStart.copy( dollyEnd );
scope.update();
break;
case 3: // three-fingered touch: pan
if ( scope.noPan === true ) return;
if ( state !== STATE.TOUCH_PAN ) return;
panEnd.set( event.touches[ 0 ].pageX, event.touches[ 0 ].pageY );
panDelta.subVectors( panEnd, panStart );
scope.pan( panDelta.x, panDelta.y );
panStart.copy( panEnd );
scope.update();
break;
default:
state = STATE.NONE;
}
}
function touchend( /* event */ ) {
if ( scope.enabled === false ) return;
scope.dispatchEvent( endEvent );
state = STATE.NONE;
}
this.domElement.addEventListener( 'contextmenu', function ( event ) { event.preventDefault(); }, false );
this.domElement.addEventListener( 'mousedown', onMouseDown, false );
this.domElement.addEventListener( 'mousewheel', onMouseWheel, false );
this.domElement.addEventListener( 'DOMMouseScroll', onMouseWheel, false ); // firefox
this.domElement.addEventListener( 'touchstart', touchstart, false );
this.domElement.addEventListener( 'touchend', touchend, false );
this.domElement.addEventListener( 'touchmove', touchmove, false );
window.addEventListener( 'keydown', onKeyDown, false );
// force an update at start
this.update();
};
THREE.OrbitControls.prototype = Object.create( THREE.EventDispatcher.prototype );
theta-view.js
var theta_view = function (video_id) {
var scene = new THREE.Scene();
var width = 600;
var height = 400;
var fov = 60;
var aspect = width / height;
var near = 1;
var far = 1000;
var camera = new THREE.PerspectiveCamera( fov, aspect, near, far );
camera.position.set( 0, 0, 0.1 );
var renderer = new THREE.WebGLRenderer();
renderer.setSize( width, height );
var element = renderer.domElement;
document.body.appendChild( element );
var directionalLight = new THREE.DirectionalLight( 0xffffff );
directionalLight.position.set( 0, 0.7, 0.7 );
scene.add( directionalLight );
var video = document.getElementById( video_id );
var texture = new THREE.VideoTexture( video );
texture.minFilter = THREE.LinearFilter;
texture.magFilter = THREE.LinearFilter;
texture.format = THREE.RGBFormat;
var geometry = new THREE.SphereGeometry(100, 32, 32, 0);
geometry.scale(-1, 1, 1);
var faceVertexUvs = geometry.faceVertexUvs[ 0 ];
for ( i = 0; i < faceVertexUvs.length; i ++ ) {
var uvs = faceVertexUvs[ i ];
var face = geometry.faces[ i ];
for ( var j = 0; j < 3; j ++ ) {
var x = face.vertexNormals[ j ].x;
var y = face.vertexNormals[ j ].y;
var z = face.vertexNormals[ j ].z;
if (i < faceVertexUvs.length / 2) {
var correction = (x == 0 && z == 0) ? 1 : (Math.acos(y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
uvs[ j ].x = x * (404 / 1920) * correction + (447 / 1920);
uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080);
} else {
var correction = ( x == 0 && z == 0) ? 1 : (Math.acos(-y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
uvs[ j ].x = -1 * x * (404 / 1920) * correction + (1460 / 1920);
uvs[ j ].y = z * (404 / 1080) * correction + (582 / 1080);
}
}
}
geometry.rotateZ(-Math.PI / 2);
var material = new THREE.MeshBasicMaterial( { map: texture } );
var mesh = new THREE.Mesh( geometry, material );
scene.add( mesh );
// PCで閲覧時にマウスドラッグで操作
var controls = new THREE.OrbitControls(camera, element);
controls.rotateUp(Math.PI / 4);
controls.noPan = true;
( function renderLoop () {
requestAnimationFrame( renderLoop );
renderer.render( scene, camera );
} )();
};
Editor’s Note:
I believe that the original author’s name of mechamogera is connected to the movie Godzilla vs. SpaceGodzilla, M.O.G.U.E.R.A. (Mobile Operations G-Force Universal Expert Robot-Aero-Type) was a UN-built mecha designed to fight Godzilla following the failure of G-Force’s Mechagodzilla.
