Gyroscope and Accelerometer sensors in THETA Z1

Hi I would like to create plugin for THETA Z1 and want to use gyroscope and accelerometer sensors. But I am not sure does camera have that sensors and can I use them. Did you have experience to use these sensors for THETA Z1?

1 Like

The camera has both.

This one shows orientation

Please read the thread. it may be difficult to accurately calibrate the orientation. Please post the results of your test.

1 Like

I would like to write plugin for THETA Z1 using accelerometer and gyroscope. But I didn’t find information about that sensors in docs. As Android OS is installed on the camera I assume that standard Android APIs will work. Am I right?

1 Like

Use the Android Camera (the deprecated one) not the newer camera2. See full code examples above. Feel free to post more questions about plug-in development. @jcasman and I will be happy to try and help.

1 Like

Hi @craig thanks for response. Sure I will post the results

1 Like

Hi @Vigen_Baghdasaryan. We also need to get data from the gyroscope. Tell me, have you come to a solution?

Ave you seen this article?

I just updated the article with a link to the GitHub repo for the project.

We are writing an application for ios and android for users and we use our own, manual stitching and work with raw data. And for the correct display of the final image, we need to understand how the camera was positioned. Asking users to install a plugin is certainly an option. But is it possible to somehow get the gyroscope data directly? If there is a gyroscope in the camera, why not provide this data to the API?

1 Like

@ravil, I agree with you that it would be better if the THETA WebAPI exposed the sensor data directly. It does not. Sensor data for accelerometer and temperature are not available through the WebAPI. I will add your note to a meeting that @jcasman is going to have with our sponsor, RICOH. However, I do not believe that this will get added to the API interface. Even if it does, it will not be in time to help your project. Keep in mind that I do not work for RICOH and that this is my best guess.

Here is what you can work with right now, today.

  • gyroscope data is available from the plug-in and there is sample code to access the gyroscope data
  • the plug-in can access the WebAPI from within the plug-in. The plug-in accesses an internal web server at localhost (the camera is localhost) and can run the WebAPI
  • the plug-in can run another HTTP server inside the plug-in to expose a HTTP API you write yourself

While it is theoretically possible to do what you want to do, it is a big job.

It would be better if RICOH or the community provided a “WebAPI relay” that took the internal WebAPI calls to localhost inside the plug-in and then exposed the WebAPI through the plug-in with Wi-Fi to mobile devices.

Thank you for providing information on your requirements. This helps us to communicate with RICOH. We send weekly and monthly reports to RICOH management on community activity. Feedback from the community does influence product direction. However, the process takes time.

2 Likes

Thanks for the detailed answer. Then we will use the plugin and ask users to install it as well.

This question came up before for commercial use. I asked an experienced plug-in developer about the best course of action. I received a response in English here.

I’ll include the response below for convenience.

If you want to use THETA in AP mode, set up a web server in the THETA plugin.
Although the IP address and port number will be changed, there is an example in which the
genuine THETA Web API and the command defined by the THETA plug-in creator are mixed.
The server of the THETA plug-in acts as an intermediary.

reference
GitHub - theta-skunkworks/theta-plugin-extendedpreview: It is a sample program that is the basis when performing image processing for live-preview with THETA plug-in. This example is useful when executing frame image processing in THETA and checking the result on WebUI.

When connecting THETA in CL mode, sensor data can be sent to the outside by performing
communication that depends on the connection destination.
Once you have a path for your data, you are free to use any specific protocol. It is up to the
implementer to decide.

As I mentioned, it looks like a big job, but it seems possible to do it. The developer above has some sample code that shows the basic strategy. It may be enough.

Let us know if you get any success.

Ok, I will write here when we have any solutions.

2 Likes

@ravil I talked with our RICOH contact today. Here pointed me to this information:

The Z1 and V do top/bottom orientation correction, so gyro and accelerometer parameter pitch, roll and heading information, shown in the XMP Gpano sections of the metadata, are zeroed out.

You can set the option _top_bottom_correction to Disapply.

https://api.ricoh/docs/theta-web-api-v2.1/options/_top_bottom_correction/

In this case, then you have to do orientation yourself after that.

Hi @ravil, yes, camera has the sensors, and we get the data with the plugin. We did computation on in camera, created tcp connection between android phone and camera, and pass results of the computation to the phone with tcp connection. There were 2 challenging parts for me:

  • try to connect phone and camera (chose here WiFi direct), with minimal user interaction
  • choosing tcp connection was not good choice, because it’s hard to detect peer disconnection (you will need to ping periodically, in both side, to make sure that peer connected), so I’ll suggest to use high level applications, like HTTP connection, or web socket
2 Likes

Wow. First, congratulations on implementing this sophisticated solution. This is the first documented success of this technique for business use. Second, thank you for the advice on using HTTP or WebSocket.

In the last six months, there’s been a surge of interest in experimenting with the IMU data for commercial use. I’m not sure if the current health-related in-person restrictions accelerated trends or if the technology of using 360 images was already advancing and had reached a tipping point. There’s a lot of development being done for workflow optimization and you’re right at the forefront of the trend.

Congratulations again on getting the technology working in a commercial setting. Wonderful.

1 Like

Vigen, another idea. If ravil’s company does not compete in the same market as your company, is there any possibility of cooperating on the same base library you built that handles getting the sensor data and communicating it your mobile phone app?

It may not be possible for intellectual property reasons. However, if it’s possible, it may help with testing the reliability of the process to get sensor data and send it to a mobile phone.

Hi @craig, thank you for interest.

yes, intellectual property doesn’t belong to me, so can’t answer to this :slight_smile:. In other side library will need to be reviewed before publication, because currently it bounded to the messages we use and seems now it’s not time for that :grinning:. So I suggest to discuss issues and solutions here (in community channel). I’ll be glad to help.

p.s. we didn’t pass the measured data to the phone, instead we did calculation in plugin (camera) and pass results to the phone

2 Likes

If @Vigen_Baghdasaryan and @ravil have time, please fill out the plug-in developer survey.

I believe that registered plug-in developers received an email to fill out a survey. I filled it out myself as a normal plug-in developer. My example responses are here.

In the past, I have heard from the community about the suggestions below:

  • want to publish plug-in privately only for company staff and have it be easily installed through the RICOH plug-in store on all Z1 devices. Does not require device to be unlocked.
  • want to have immediate plug-in publication feature for “test” versions during development when you send an updated plug-in to your staff for internal testing (similar to Google Play testing process). Does not require camera to be unlocked.
  • want to develop plug-ins or install with adb more widely within company, but the plug-in program invalidates camera warranty and company does not allow this. Want to install my own private plug-in easily on unlocked devices and develop without invalidating my warranty.
  • want to have a mechanism for customers to pay for plug-in, similar to Google Play
  • want to have ability to issue software license key for plug-in to limit use only to paying customers
  • want to connect plug-in to another device with the USB port (such as Arduino, Raspberry Pi, or other Linux SBC) and the USB device cannot be accessed unless I use Vysor to allow use of the USB device

If even one of the issues is relevant to you, please fill out the survey. Although we’ve been sending suggestions to RICOH, it is much more powerful if the developer sends it directly.

Thank you for your help.

1 Like

Hi @Vigen_Baghdasaryan and @craig has there been an update on this. Can is it possible to fetch accelerometer data now from Z1?

You can get the accelerometer data from the plug-in, but there is no WebAPI to access the accelerometer data.