Theta X for mapping. Can it do better in positioning quality?

Ricoh Hardware and Software
1

Copy from my post in the Facebook group
(10) Ricoh THETA 360° Camera User Group | # Theta X for mapping | Facebook

[updated 2023-09-06]

I would like to draw attention to this subject and offer some thoughts that may help to improve position accuracy. I have addressed image quality before. Disclaimer: I am not a GNSS specialist, an informed user only.

Mapping with 360 imagery, Google Street View and Mapillary that is, aids: e.g., map creation; digital twin creation; travel destination promotion; city and infrastructure planning; disaster recovery; access, value, risk, damage assessment. It can be regarded as part of the Architecture, Engineering and Construction field of profession, as well as marketing, insurance, rescue services. Position precision and image quality are both key in this. Contrary to what Ricoh states in its blog Reach the full potential of RICOH THETA X with built-in GPS l RICOH360 Blog, position accuracy should be of navigation quality, which is not within reach of the current GNSS module.

First off, I think for this mapping use case we would better have a dedicated plug-in to experiment and develop proper dedicated functionality without overloading the general settings with options. Professor Taichi Furuhashi suggests the same in Ricoh360 blog The Potential of RICOH THETA X Built-in GPS Function - Interview with Professor Furuhashi, Aoyama Gakuin University l RICOH360 Blog.

Currently, the most convenient way to capture for mapping are the 8K 2 and 10 fps video modes, which includes the Google defined Camera Motion Metadata, CaMM track in the video file; Camera Motion Metadata Spec | Street View Publish API | Google for Developers. Theta X writes the case 2, 3 and 5 type data packets. I think it may be useful to write a case 6 type GPS information data packet instead of the case 5 type and add a case 7 type with the compass data.

In the blogs Ricoh expresses ‘The ability to acquire location information at short intervals during low frame rate video is one of the THETA X abilities that we hope will be utilized by construction and other industries.’. I think it is too much of an expectation that 3rd parties and users will each write code to extract video frames and calculate proper position for each from the CaMM. Dean Zwikel’s toolset helps in this. He can be contacted through Facebook Messenger.

Inspection of the Theta X CaMM track has shown that relative within the file, the first position data is timed about 0.8 seconds after the first video frame. As far as I know, video frame track and position data track are best be synchronized in time, which doesn’t seem to be the case here. Video frames in time before the first position data package will have to be dropped in the processing.

Theta X uses 1 Hz frequency only in the GNSS module to acquire position. However, it offers video frame rates of 2 and 10 fps.

Google guidelines for capturing are as follows.

  • Under 5 mph or 8 km/h for 1 FPS; walking.

  • Under 30 mph or 45 km/h for 5 FPS; bicycle, light motorcycle.

  • Under 45 mph or 70 km/h for 7 FPS; car, motorcycle.

Walking 2 fps, say I take 1 step of 1 meter each second, then we have a GNSS position for every meter covered and every other frame.

80 km/h 10 fps = 22 meters/sec 10 fps, then we have a GNSS position for every 22 meters and for 1 out of the 10 frames only. I think that in curves or with quick speed changes this leads to a position deviation that can be resolved by setting the GNSS module to a higher position acquisition frequency. That requires switching the GNSS module from the low power Super-E mode to the regular Continuous mode. The Continuous mode consumes more power, but I consider that no longer of the essence now that we can power the Theta with a powerbank.

ZOE-M8B_ProductSummary_(UBX-17012173).pdf (u-blox.com)

ZOE-M8B (u-blox.com)

The GNSS module needs to retain locks on the moving satellites while moving itself. The camera is not just moving forward, but swaying as well, depending on the transport mode. The GNSS module has settings to take this into account for performance and accuracy. Relevant settings are Pedestrian, Automotive, Bike. Refer paragraph 8.1 Platform settings and paragraph 32.10.19.1 Navigation engine settings in u-blox8-M8_ReceiverDescrProtSpec_UBX-13003221.pdf.

So, the user interface needed would have the following modes, avoiding separate settings.

  • pedestrian: video 8K 2fps, GNSS platform set to Pedestrian, Continuous mode (refer below on Galileo) 1 Hz refresh.

  • bicycle, light motorcycle: 8K 5fps, GNSS platform set to Bike, Continuous mode 3 Hz refresh.

  • motorcycle: 8K 10fps, GNSS platform set to Bike, Continuous mode 5 Hz.

  • car: 8K 10fps, GNSS platform set to Automotive, Continuous mode 5 Hz.

The frequencies are my educated guess, but I gladly leave that to the specialist to determine.

Last, but not least, the number of GNSS’ used simultaneously. It should be noted that now both BeiDou and Galileo are fully operational and generally deliver a higher precision than GPS and GLONASS. General rule of thumb is, that the more GNSS’ are used simultaneously, the better the positioning and less risk of having insufficient clear line of sight satellite locks. This is important especially in nature or city canyons. A test posted earlier by Sam Rohn in New York demonstrates serious position issues with current settings. The GNSS module is likely running in a manufacturer default mode with Super-E power saving and GPS, GLONASS only, while it can use 3 GNSS’ simultaneously. As power consumption while using a power bank is no longer an issue, I suggest improving by using Continuous mode and 3 of the GNSS’ always. Which 3 can automatically be determined by where on Earth the capture is and which 3 GNSS’ are most effective there, taking into account availability of support systems like SBAS too. I would definitely appreciate the use of Galileo.

That is it for now and I hope this helps to improve Theta X’s effectiveness and usability.

1 Like
2

@jcasman if you want to summarize the main points and put into the agenda, you may want to reference the Camera Motion Metadata Spec for this section:

Theta X writes the case 2, 3 and 5 type data packets. I think it may be useful to write a case 6 type GPS information data packet instead of the case 5 type and add a case 7 type with the compass data.

To @jcasman I have not checked this

Inspection of the Theta X CaMM track has shown that relative within the file, the first position data is timed about 0.8 seconds after the first video frame. As far as I know, video frame track and position data track are best be synchronized in time, which doesn’t seem to be the case here.

You could attempt to verify and then put information in this document. I’ll change the document title from GPS to GNSS.

Google guidelines for capturing are as follows.

  • Under 5 mph or 8 km/h for 1 FPS; walking.
  • Under 30 mph or 45 km/h for 5 FPS; bicycle, light motorcycle.
  • Under 45 mph or 70 km/h for 7 FPS; car, motorcycle.

framerates are here theta-api-specs/theta-web-api-v2.1/options/file_format.md at main · ricohapi/theta-api-specs · GitHub

The 5fps may be newer?

@jcasman you’ll need to summarize the 2nd half of the article.

3

@craig, @jcasman

I have that 0.8 sec time lag from an inspection by @Dean_Z.
Z1 used to have an issue with timing too. I do not not know if it ever got resolved. @Shotaro_Igami knows.

Those Google frame rate recommendations have been there all along in their Help system. The lower frame rates just avoid an excess of imagery, taking the camera movement speed into account. They save on needed stitching, battery, storage, upload and Google processing.

That year ago @Sam_Rohn position quality test was walking along a sidewalk in New York. The trace showed many anomalies. Likely caused by buildings on one hand blocking signals and signal reflections from buildings too. The usual city canyon issue. Of course those situations exist in mountainous nature too.

I feel camera manufacturers should get more serious on selection of antenna and fused GNSS / Inertial Measurement Unit module, as well as the tuning thereof, exposing usefull settings too.

4

@JoscTr Thanks for posting. I’m summarizing the details and will provide the information to a RICOH manager today. Just to be clear, I’m not suggesting it’s a solution. But hopefully we’re improving communications.

We work with RICOH to help build the community, we can:

  • Summarize your points (from multiple posts over time) to make it easier for RICOH to understand
  • Organize information into central document. We’ve already started improving it based on your information.
  • Discuss with product manager
  • Include the information in a written report to a wider group of managers, including engineering managers

To be clear:

  • We are unable to discuss directly with the engineer in charge of GNSS development
  • At this time, we will not communicate directly with the engineering team, only through the managers
2 Likes
5

@JoscTr first, thank you for all your help. The GNSS information is new to us.

I have made changes to this document, which is freely available to developers:

  1. link to your article on this forum
  2. changed title to IMU and GNSS Sensors (was GPS). I am going through the document to find other uses of GPS and will change to GNSS
  3. reference to u-blox ZOE-M88 GNSS module based on u-blox press release

I will include the changes in the meeting agenda with RICOH and monthly community summary report to management.

I believe that @jcasman is working on a more detailed summary of your article.

In addition to your valuable feedback, our overall strategy is to get more feedback from companies that want to use the GNSS technology in the THETA X. We are using the document linked above to send information to developers in companies.

The combination of community and industry feedback may help RICOH in future product decisions.

1 Like
6

Sounds good @craig . I think Sam ones told me that even if we have some developers with whom we can talk directly, they cannot do anything without orders from managers.

It seems to me that a first step to improve on this is that developers create the fundamental api’s. Then Ricoh or someone with the skills could build a dedicated plug-in with easy settings combining api calls. There are more requirements for such a plug-in than the current topic.

I understand that Ricoh has a renewed focus on Construction, Architecture, Engineering and so. I think it helps to realize that “(3D) mapping” and creating “digital twins” is part of this and that position quality is a key quality aspect. Refer CupixVista | AI, 360° Photogrammetry, 3D Map.

My comment about a fused GNSS / IMU (Inertial Measurement Unit) is because such a fused uBlox module will use both technologies to determine position. If GNSS signal is lost, then it will be able to provide correct position for some covered distance. This is relevant for passing under bridges, tunnels etc. Even if there is sufficient GNSS signal, it will help to improve position. Of course this is a matter for future hardware.

(As for image quality, 8K is fine for now and there are sufficient other usual camera settings. I would appreciate an improvement in dynamic range. 8K 60 fps video capability would be nice for non mapping use. However, looking at how Insta360 is moving, I imagine Ricoh might want to stretch specs. I do not feel it to be necessary to stitch in real-time or on device at all if that would help realizing 11K video for mapping.)

1 Like
7

@craig, here a screen shot of an old Facebook post of mine. It shows positioning anomalies with X. The capture was on foot, mounted on helmet. The anomalies occur when I had to slow down or stop and wait.

image
image908×702 172 KB

8

Following screen shot speaks for itself.

image
image656×1025 187 KB

9

@JoscTr thanks for this information. I had not seen the interview before.

The Potential of RICOH THETA X Built-in GPS Function - Interview with Professor Furuhashi, Aoyama Gakuin University

Hopefully, as the THETA X is used more widely in the GIS and related areas, software fixes can be applied.

I’ll summarize the points as follows:

  1. erroneous data sometimes occurs in Mapillary when the there are pauses or changes to a straight line, constant pace
  2. request for option to prevent recording 8K or 5.7K low fps video (with camm GNSS data) unless the GNSS chip has a more stable satellite lock.

Another thing that I didn’t fully appreciate is that for A-GPS to work, the THETA X must first be connected to a WiFi router in client mode.

Thanks.

10

About your A-GPS, Assisted GPS, remark. Refer here Assisted GNSS - Wikipedia

On 1. The erroneous data does not occur in Mapillary. It is erroneous positioning by X. In this case Google Street View rejecting it (left pic). Mapillary accepts as recorded (right) pic, which explains the issue, showing unstable positioning at standstill.