HowTo: Develop 360 Image Desktop Apps with JavaScript and RICOH THETA

javascript

#1

It’s easy to develop desktop applications that control the RICOH THETA API. This tutorial shows you how to use JavaScript, Node, and Electron to build applications that run on Mac, Windows, and Linux laptop and desktop computers. Desktop applications are useful for commercial applications like building surveillance monitoring or automobile sales.

I’ll show you how to get started with Electron, but it’ll be up to you to finish the application and build your million dollar VR/AR business. :slight_smile: :theta:

Learning Objectives

  • Create template desktop application framework
  • Send and receive data with the THETA using WiFi
  • Start Session, Set API to v2.1 (the latest)
  • Take picture
  • Download picture from camera to laptop
  • Display picture

Install Node

Go to nodejs.org and install node. I used the LTS version.

Install Electron

npm install electron -g

Create index.html and app.js

Note: working example is available on GitHub.

Create a new folder for your project.

mkdir electron-tutorial

Change directory into the folder.

cd .\electron-tutorial\

Make two files, index.html and app.js

index.html

 <!DOCTYPE html>
 <html>
 <head>
 </head>
 <body>
   <h1>My RICOH THETA VR Application</h1>
 </body>
 </html>

app.js

 var electron = require('electron')

 electron.app.on('ready', function () {
   var mainWindow = new electron.BrowserWindow({width: 600, height: 800})
   mainWindow.loadURL('file://' + __dirname + '/index.html')
 })

Tip: both atom from GitHub and VS Code from Microsoft are built with Electron and are great editors for JavaScript.

Test Application

In a command line terminal, run electron app.js

electron app.js

You will see a new window open with your basic application.

Congratulations, you’ve just developed a basic desktop application with Electron! Yay! :tada: :theta:

Initialize Electron Package

npm init

You will see a text-based wizard. Press return and accept all the defaults.

After going through the wizard, you will have a new package.json file.

Install node-rest-client

npm install node-rest-client --save

Create index.js

index.js

require('node-rest-client');

That’s it. The file is just one line right now.

Add script to index.html

In your index.html file, add this after the <h1> section.

 <script type="text/javascript" src="index.js"></script>

Run electron app.js again and make sure your app is still running. There should be no change.

Tip You can access developer tools to debug your application

Add Response Area

In index.html, add these lines above the <script> tag.

<h2>Response</h2>
<p id="thetaResponse"></p>

Grab THETA Info with HTTP

In index.js, add these lines:

require('node-rest-client');
var fs = require('fs');
var request = require('request');

var Client = require('node-rest-client').Client;

var client = new Client();


getInfo = function() {
    console.log("button clicked")
    client.get("http://192.168.1.1:80/osc/info", function (data, response) {
      console.log(data);
      // console.log(response);
    var thetaResponse = document.getElementById('thetaResponse');
    thetaResponse.innerHTML = JSON.stringify(data);
    });
  }

Build Button

In index.html, add these lines:

 <h2> Testing and Info</h2>
 <button onclick="getInfo();">Get THETA Info</button>

Connect Your Computer to THETA

Connect your computer to the THETA with WiFi. If you are unsure about this step, refer to the Unofficial API Guide.

Test Application

electron app.js

Press the button.

Congratulations! You’ve successful built a desktop application for your RICOH THETA. You’re awesome!

Extend the Application

You’re off and running. You should now be able to extend the application with other API commands to take a picture and download it to your laptop. Numerous applications exist to display the 360 image. Search through this site for ideas, or ask a question below.

Here’s a GitHub repository to an application with a bit more features.

Reference

More examples

If you’d like the challenge of exploring POST commands to the THETA as well as figuring out how to do file downloads, skip the sections below. You can also jump straight to the GitHub repository

HTTP POST Commands

The data from the /osc/info command is obtained with an HTTP GET command with no arguments. For most other THETA API commands, you’ll need to use POST.

Example:

startSession = function() {
    var args = {
    data: { "name": "camera.startSession" },
    headers: { "Content-Type": "application/json" }
};
    client.post("http://192.168.1.1:80/osc/commands/execute", args, function (data, response) {
      console.log(data);
    var thetaResponse = document.getElementById('thetaResponse');
    thetaResponse.innerHTML = JSON.stringify(data);
    });
  }

The example above uses client.post, not client.get. You must also pass the POST command arguments.

Setting API to v2.1

There are two versions of the RICOH THETA API, v 2.0 and v 2.1. I’m using v2.1 in this example. v2.1 doesn’t require the use of sessionId.

setApiV2 = function() {
    var args = {
    data: { "name": "camera.setOptions",
          "parameters":{
            "sessionId": "SID_0001",
            "options": {
              "clientVersion": 2
            }
          } 
        },
    headers: { "Content-Type": "application/json" }
};
    client.post("http://192.168.1.1:80/osc/commands/execute", args, function (data, response) {
      console.log(data);
    var thetaResponse = document.getElementById('thetaResponse');
    thetaResponse.innerHTML = JSON.stringify(data);
    });
  }

Take Picture Example

This example uses the v2.1 API.

takePicture = function() {
    var args = {
    data: { "name": "camera.takePicture"},
    headers: { "Content-Type": "application/json" }
};
    client.post("http://192.168.1.1:80/osc/commands/execute", args, function (data, response) {
      console.log(data);
    var thetaResponse = document.getElementById('thetaResponse');
    thetaResponse.innerHTML = JSON.stringify(data);
    });
  }

Image Download Example

There are many ways to download the image. In this example, I am using request. Although you can grab the _latestFileUri from
state, I am using
listFiles as you an use the same snippet to list the files on your camera.

 getImage = function() {
    var lastImageUrl;
    var args = {
      data: {
        "name": "camera.listFiles",
        "parameters": {
          "fileType": "image",
          "entryCount": 1,
          "maxThumbSize": 0
        }
      },
      headers: {"Content-Type": "application/json"}
    }
    client.post("http://192.168.1.1/osc/commands/execute", args, function (data, response) {
     lastImageUrl = data.results.entries[0].fileUrl;
       var download = function(uri, filename, callback){
    request.head(uri, function(err, res, body){
     console.log('content-type:', res.headers['content-type']);
     console.log('content-length:', res.headers['content-length']);

    request(uri).pipe(fs.createWriteStream(filename)).on('close', callback);
  });
};
    console.log(lastImageUrl);
    // download('http://192.168.1.1/files/744a605553442020024b0202cb00f201/100RICOH/R0012006.JPG', '360_images/lastFile.jpg', function(){
    download(lastImageUrl, '360_images/lastFile.jpg', function(){

    });
  });
  }

TIP: there are two separate steps. 1) get the URL of the last file on the camera, 2) download the file and save to local storage on your laptop. For debugging purposes, you can hard code a known file on your camera and test the download separately from your code to get the last file URL.

Viewing a THETA 360 Image in Electron

The sample app uses Google VR View for Web to display the 360 image and provide navigation. Another idea is to use A-Frame.

You can resize or apply filters to the RICOH THETA image. You must preserve a ratio of 2:1. For information on color, resize, metadata, orientation and tilt, see the THETA Media Unofficial Guide.

In index.html add this line

  <iframe width="100%" height="400px" allowfullscreen
frameborder="0" src="googlevr/googlevr.html?image=../360_images/lastFile.jpg&is_stereo=false"></iframe>

The parameters for Google VR View can be found on the Google site.

In the sample application, you’ll need to grab the folder for googlevr.

You do not need to edit anything. Though, if you open googlevr.html, you can edit some parameters.

360 Video

THETA video works great in Electron.

Just change index.html to contain something like this:

<iframe width="100%" height="400px" allowfullscreen
frameborder="0" src="googlevr/googlevr.html?video=../360_images/video_sample.mp4&is_stereo=false"></iframe>

Note that I’m using video= and not image=

Sound works. As the test was done with the RICOH THETA S, I have not tested spatial sound. Will test that when the next version of the THETA comes out.

request to download image.

I’m using request to download the image from the camera to the laptop. I have not checked to see when the image has completed downloading. This would be a great next step. Once the image has downloaded, then refresh the image window.

Next Steps

  • Set up an auto-refresh with location.reload() or similar technique

How to read the 360 degree image in Raspberry Pi
Proper steps for working on Theta API v2.1
#2

I added a sample THETA 360 video clip to the example application on GitHub.

Compared to YouTube, the resolution seems higher to me. I think YouTube compresses the video for streaming and may lose some resolution.

The other advantage of storing the videos locally inside of the application is that there are no network congestion problems.

If you have an application with specific videos like industrial training of a jet engine or a factory tour for new employees, it may be better to put the video file inside of the application and store it locally.

It’s also good if you’re at a remote area or have low bandwidth.


#3

I wouldn’t be surprised if there was some video quality loss when you upload to YouTube. Also no network issues. That makes sense.


#4

Using this tutorial as a starting point, community member wrote new Electron application with streamlined workflow and interface and tested it in the equipment room of a famous Silicon Valley University. If you build an app, post the code or a screenshot. It’ll give all of us more ideas.


#5

To save your self some time when testing your Application…

  1. open notepad
  2. paste in “electron app.js”
  3. save as “RUN.cmd” in the same location as your application (…\electron-tutorial)

now all you have todo is open “RUN.cmd”, instead of typing or pasting “electron app.js” everytime you want to test.


#6

Im Back at it again…
but i cannot connect to my theta v in client mode,
using your electron-test on gethub example.

i changed the ip address in index.js to use the new address.
and tryed just going to http://NEWipADDRESS/osc/info
and it asks for username and password.
i tryed “THETAYLXXXXXXXX”, passwd is “XXXXXXXX” no luck.

any ideas? @codetricity


#7

@Kev091190

Using client mode can take some fussing, but once you get it, it should work well.

Have you updated your firmware? That might help. You want 2.50.1

Here’s a fuller guide: THETA V Client Mode Configuration Guide


#8

yes im currently on 2.50.1
ive already connected it successfully to my “router” (my seagate wireless plus HDD that also has its own client mode so it is connected it the internet aswell)

Green light is solid


#9

Check out from Step #4 in the link I added. In the official mobile app you can set up the username and password.

Yea, you’ll want to connect directly to the THETA for the first setup.


#10

i did set up a password, but re entered a new password and now i can connect using the internet browser.

next what code would i have to add to allow the electron app to use the password and username that was set?.


#11

i got it now

Redirect 192.168.1.1 to new address

http://IPADDRESS/osc/info?username=THETAYLXXXXXXXXX&password=NEWPASSWORD

---- [edit] incorrect code below-------
example: (from index.js file)

getInfo = function() {
    console.log("button clicked")
    client.get("http://IPADDRESS/osc/info?username=THETAYLXXXXXXXXX&password=NEWPASSWORD", function (data, response) {
      console.log(data);
      // console.log(response);
    var thetaResponse = document.getElementById('thetaResponse');
    thetaResponse.innerHTML = JSON.stringify(data);
    });
  }

Thank you very much for the help


#12

Hey, fantastic. Good work!


#13

Thanks for posting the code snippet, too. Nice to get that out there for others.


#14

my mistake. it will allow you to access it through the web browser but not through the electron app

ill keep trying :confused:


#15

im guessing it would have to be something of this sorts? (in arrows below)
but it does not exist in the api v2.1

startSession = function() {
    var args = {
    data: { "name": "camera.startSession" , 
      ------> "client" : {
                              "username" : "THETAYLXXXXXXXXX" ,
                              "password" : "NEWPASSWORD"
                               }    <-----------------
 },
    headers: { "Content-Type": "application/json" }
};
...

#16

The client mode user name and password is with digest authentication.

I have not tried the library below:

I think the request library below does not implement digest auth correctly.

May Not Work


You Need a Library That Handles DigestAuth

var postData = {
// post data here
}

var postReq = new digestAuthRequest('POST', url, 'username', 'password');

postReq.request(function(data) { 
  // success callback
  // data probably a success message
},function(errorCode) { 
  // error callback
  // tell user request failed
}, postData);

#17

ill keep working on this after im done my current project.

To use the current code to make a simple interface that takes video or pictures and back them up to a harddrive when there done, possibly try to do long video recording then transfer and loop.
i have a car show i go to id like to record since i foundout you can inport 360 video into a 3d render program i have.

.
A little improvement code: (to update on to my previous CMD notepad edit)
Using the “application on GitHub”
Features:

  • Adds Close Program Button to web page
    .thetalist
  • Also minimizes the “RUNTHETA” program at start

    ’ note i did add “hidden” to the image and video demo(s)

Step A - Create:

  • "RUNTHETA.bat" (used to run program)
    open notepad & paste code below, then save as RUNTHETA.bat

if not DEFINED IS_MINIMIZED set IS_MINIMIZED=1 && start "" /min "%~dpnx0" %* && exit TITLE RUNTHETA electron app.js exit

  • "KILLRUN.bat" (used to Close RUNTHETA.bat also Close’s Electron)
    open notepad & paste code below, then save as KILLRUN.bat

taskkill /F /FI "WindowTitle eq RUNTHETA" /T


Step B - Add:

  • Add The Button to your App Page " X " & “Refresh Page” Button
    open “index.html” in notepad & paste code below where you’d like, then save.

<button onclick="KILLRUN();">X</button> <button value="Refresh Page" onClick="window.location.href=window.location.href">Refresh Page</button>

  • Add The Button code
    open “index.js” in notepad & paste code below at the bottom of the Document, then save.

KILLRUN = function() { require('child_process').exec('cmd /c KILLRUN.bat'); }


I will also make a powershell program when i have time that Adds this all for you…
so all you have todo it run the program.

enjoy.


#18

doing some testing…
made a easy way to input code for testing…


#19

Nice progress! Thanks for these updates. :slight_smile:


#20

More progress:
finaly after some time trying to figure this out
i found a way to find info from a result

example code (result code from a request):

{
	"fingerprint":"FIG_0019","state":
	{
		"_apiVersion":2,
		"batteryLevel":1,
		"_batteryState":"charged",
		"_cameraError":[],
		"_captureStatus":"idle",
		"_capturedPictures":0,
		"_latestFileUrl":"",
		"_pluginRunning":false,
		"_pluginWebServer":false,
		"_recordableTime":0,
		"_recordedTime":0,
		"storageUri":"http://192.168.1.1/files/150100525831424d42075931ce41c300/"
	}
}

and say i want the url from the “storageUri” everytime i use the same request and get the above result above.

well i can get that and extract only the url.
as of now ive made it so all you have todo is select the name of the element you want, and if the element result requres (Quotation marks, comma) before or after the info you want to require.


*the picture displays it getting the “fingerprint” element 1st result
also the code on the page isnt displayed the best, but i was more focused on getting the equation to work as i wanted then make it look good for myself lol