NVIDIA Jetson and USB API for RICOH THETA


NVIDIA Jetson with Jetpack 4.6 using USB API with THETA Z1 fw 2.10.3 on Oppkey robot

In the .guide community, there are two primary ways to use the USB API with NVIDIA Jetson models:

project library app pros cons oppkey opinion
libptp2 ptpcam works with all USB API commands that .guide tested. No known problems. In wide use in .guide community code has not been updated in over a decade. Project appears to have no active maintainer. Code base and library may be difficult to use for development use for testing, prototyping, small deployment
libgphoto2 gphoto2 project is under recent development and is maintained. Due to the more updated code base and modern library, may be more popular for production use. not as widely used in .guide community. May need patches. use for larger projects, long term projects, or large production

The USB API works with Z1, X, V, SC2, and S.

First, make sure that your RICOH THETA is not mounted.

As I’m on a limited budget, I do not have a Jetson Xavier. I’m testing an NVIDIA Jetson B01 with Jetpack 4.6, the most recent version of JetPack for the Nano. On Xavier, people may be using JetPack 5.0. Unfortunately, I can’t test against JetPack 5, but I think this process will work.

What you need

basic build tools

sudo apt install build-essential
sudo apt install libtool
sudo apt install automake

libusb 1.0.21 development library in libusb-1.0-0-dev

  • libusb-1.0-0-dev (may already be installed. This tutorial uses libusb-1.0.21 on JetPack 4.6)
    • libusb-1.0-0-dev_1.0.21-2_arm64.deb
    • if you have old usb libraries installed, uninstall the libraries
      • remove libusb-0.1-4_0.1.12-31_arm64.deb
      • remove libusb-dev_0.1.12-31_arm64.deb

NOTE: this should also work with the libusb 1.0.26 version that is the most recent version as of Nov 2022. If you have problems with libusb 1.0.26, please post and I’ll install it from source to test it with a Z1.

libptp2 option

  1. libptp2 src
  2. libptp2 patches
  3. libusb-compat-0.1

I have forked the patches from community member nickel110 and added a README that is specific to RICOH THETA:

libgphoto2 option

I’ve forked libgphoto2 and applied the patches in this repo:

show version of JetPack

Jetpack 4.6.

$ apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.6-b199
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b199), nvidia-opencv (= 4.6-b199), nvidia-cudnn8 (= 4.6-b199), nvidia-tensorrt (= 4.6-b199), nvidia-visionworks (= 4.6-b199), nvidia-container (= 4.6-b199), nvidia-vpi (= 4.6-b199), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)

Technique 2 to find JetPack version

 cat /etc/nv_tegra_release 
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021

You can confirm the JetPack version with the archive information.

JetPack Archive | NVIDIA Developer

This corresponds to Ubuntu 18.04

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

libusb version

These are the versions shown prior to installing the compatibility layer.

 ls /usr/lib/aarch64-linux-gnu/*usb*
/usr/lib/aarch64-linux-gnu/libgusb.so.2
/usr/lib/aarch64-linux-gnu/libgusb.so.2.0.10
/usr/lib/aarch64-linux-gnu/libusb-1.0.a
/usr/lib/aarch64-linux-gnu/libusb-1.0.so
/usr/lib/aarch64-linux-gnu/libusbmuxd.so.4
/usr/lib/aarch64-linux-gnu/libusbmuxd.so.4.0.0

libusb-compat v0.1.8

libusb-compat-0.1 v0.1.8 was released on November 18, 2022.

As libptp2 looks for an older version of libusb, I am using this helper library to use libusb-1.0 with libptp2. If you do not want to install libusb-compat, consider using libgphoto2 instead of libptp2.

run autogen.sh, configure, make.

ls
aclocal.m4      config.h       depcomp           libusb.pc     missing
AUTHORS         config.h.in    examples          libusb.pc.in  NEWS
autogen.sh      config.log     INSTALL           LICENSE       README
autom4te.cache  config.status  install-sh        ltmain.sh     stamp-h1
bootstrap.sh    config.sub     libtool           m4
ChangeLog       configure      libusb            Makefile
compile         configure.ac   libusb-config     Makefile.am
config.guess    COPYING        libusb-config.in  Makefile.in
craig@jetpack-4:~/Development/libusb-compat-0.1$ make
make  all-recursive
make[1]: Entering directory '/home/craig/Development/libusb-compat-0.1'
Making all in libusb

Install

sudo make install
Making install in libusb
make[1]: Entering directory '/home/craig/Development/libusb-compat-0.1/libusb'
make[2]: Entering directory '/home/craig/Development/libusb-compat-0.1/libusb'
 /bin/mkdir -p '/usr/local/lib'

run sudo ldconfig

confirm after installing libusb-compat

craig@jetpack-4:/usr/local/lib$ ls -l libusb*
lrwxrwxrwx 1 root root     19 Nov 17 10:01 libusb-0.1.so.4 -> libusb-0.1.so.4.4.4
-rwxr-xr-x 1 root root  75896 Nov 17 10:01 libusb-0.1.so.4.4.4
-rw-r--r-- 1 root root 149794 Nov 17 10:01 libusb.a
-rwxr-xr-x 1 root root    941 Nov 17 10:01 libusb.la
lrwxrwxrwx 1 root root     19 Nov 17 10:01 libusb.so -> libusb-0.1.so.4.4.4

building libptp2

libptp2-theta$ make
make  all-recursive
make[1]: Entering directory '/home/craig/Development/libptp2-theta'
Making all in src
make[2]: Entering directory '/home/craig/Development/libptp2-theta/src'
/bin/bash ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I..     -g -O2 -Wall -Wmissing-declarations -Wmissing-prototypes -MT ptp.lo -MD -MP -MF .deps/ptp.Tpo -c -o ptp.lo ptp.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -g -O2 -Wall -Wmissing-declarations -Wmissing-prototypes -MT ptp.lo -MD -MP -MF .deps/ptp.Tpo -c ptp.c  -fPIC -DPIC -o .libs/ptp.o

Inside of /src/, I name my modified version theta.

file locations

ptpcam is the unmodified libptp2 program that came with the src. theta is the version with modifications.

 ls -l /usr/local/bin/
total 312
-rwxr-xr-x 1 root root   1298 Nov 17 10:01 libusb-config
-rwxr-xr-x 1 root root 162880 Jul  5 16:41 ptpcam
-rwxr-xr-x 1 root root 147536 Nov 17 10:22 theta

get info

 ./theta --info

THETA Device Info
==================
Model: RICOH THETA Z1
  manufacturer: Ricoh Company, Ltd.
  serial number: '10010104'
  device version: 2.10.3
  extension ID: 0x00000006
  image formats supported: 0x00000005
  extension version: 0x006e

take picture

./theta --capture

Initiating capture...
Object added 0x00000091
Capture completed successfully!

list files

 ./theta --list-files

Listing files...
Camera: RICOH THETA Z1
Handler:           Size: 	Captured:      	name:
0x00000091:      8028786	2022-11-17 19:19	R0010085.JPG

additional tests

file transfer of image

 theta --get-all-files
Camera: RICOH THETA Z1
Saving file: "R0010085.JPG" is done.

test file to make sure it didn’t get corrupted.

file transfer of 2.2GB videofile

$ theta --get-all-files
Camera: RICOH THETA Z1
Skipping file: "R0010085.JPG", file exists!
Saving file: "R0010086.MP4" 
is done.
craig@jetpack-4:~/Documents/tmp$ 
craig@jetpack-4:~/Documents/tmp$ ls -l
total 2182384
-rwxr----- 1 craig craig    8028786 Nov 17  2022 R0010085.JPG
-rwxr----- 1 craig craig 2226722795 Nov 17  2022 R0010086.MP4

Delete File

theta --delete-object=0x00000092

Object 0x00000092 (R0010086.MP4) deleted.
craig@jetpack-4:~/Documents/tmp$ theta --list-files

Listing files...
Camera: RICOH THETA Z1
Handler:           Size: 	Captured:      	name:
0x00000091:      8028786	2022-11-17 19:19	R0010085.JPG

list devices and show multiple RICOH THETA cameras

theta --list-devices

Listing devices...
bus/dev	vendorID/prodID	device model
001/021	0x05CA/0x2718	RICOH THETA X
001/016	0x05CA/0x2716	RICOH THETA Z1

set to live streaming

theta --set-property=0x5013 --val=0x8005

Camera: RICOH THETA Z1
'Still Capture Mode' is set to: [Normal]
Changing property value to 0x8005 [(null)] succeeded.

set to image mode

 theta --set-property=0x5013 --val=0x0001

Camera: RICOH THETA Z1
'Still Capture Mode' is set to: 0x8005 (-32763)
Changing property value to 0x0001 [(null)] succeeded.

put camera to sleep

theta --set-property=0xd80e --val=0x01

Camera: RICOH THETA Z1
'UNKNOWN' is set to: 0
Changing property value to 0x01 [(null)] succeeded.

wake up from sleep

theta  --set-property=0xD80E --val=0

Camera: RICOH THETA Z1
'UNKNOWN' is set to: 0
Changing property value to 0 [(null)] succeeded.

Action Shot

Hobby toy robot to prototype NVIDIA Jetson control of the RICOH THETA Z1 with the USB API. Shot taken with libptp2 and JetPack 4.6.

Taken inside of TH-3 case mounted on top of robot.

Troubleshooting libptp2

An ERROR: Could not open sesson! message could indicate that your camera is mounted.

craig@jetpack-4:~$ theta --info

THETA Device Info
==================
ERROR: Could not open session!
craig@jetpack-4:~$ 

craig@jetpack-4:~$ theta --capture

Initiating capture...
ERROR: Could not open session!
craig@jetpack-4:~$ 

You can disable automount using the article below. In my tests, I needed to disable automount.

Disable THETA Automount on Linux and NVIDIA Jetson for USB API


libgphoto2

Using 2.5.30 from this fork of libgphoto2

install

autoreconf --install --symlink
./configure --prefix=/usr/local
make
make install

modifications overview

shutterspeed by Hugues

config.c (libgphoto2-2.5.30\camlibs\ptp2)

	{ N_("Shutter Speed"), 					"shutterspeed", 			PTP_DPC_RICOH_ShutterSpeed, 			PTP_VENDOR_MICROSOFT, 		PTP_DTC_UINT64, _get_Ricoh_ShutterSpeed, _put_Ricoh_ShutterSpeed },	

Line was added after this line:

{ N_("Shutter Speed"), "shutterspeed", PTP_DPC_RICOH_ShutterSpeed, PTP_VENDOR_PENTAX, PTP_DTC_UINT64, _get_Ricoh_ShutterSpeed, _put_Ricoh_ShutterSpeed },

human readable white balance values by Dahlo Park

this is git diff in libgphoto2 root dir.

diff --git a/camlibs/ptp2/config.c b/camlibs/ptp2/config.c
index 5b5e057a7..ed55ad508 100644
--- a/camlibs/ptp2/config.c
+++ b/camlibs/ptp2/config.c
@@ -1496,6 +1496,17 @@ static struct deviceproptableu16 whitebalance[] = {
        { N_("Fluorescent: Day White"), 0x8004, PTP_VENDOR_PENTAX },
        { N_("Fluorescent: White"),     0x8005, PTP_VENDOR_PENTAX },
        { N_("Fluorescent: Tungsten"),  0x8006, PTP_VENDOR_PENTAX },
+
+       { N_("Shade"),                  0x8001, PTP_VENDOR_MICROSOFT },
+       { N_("Cloudy"),                 0x8002, PTP_VENDOR_MICROSOFT },
+       { N_("Tungsten 2"),             0x8020, PTP_VENDOR_MICROSOFT },
+       { N_("Fluorescent: Daylight"),  0x8003, PTP_VENDOR_MICROSOFT },
+       { N_("Fluorescent: Day White"), 0x8004, PTP_VENDOR_MICROSOFT },
+       { N_("Fluorescent: White"),     0x8005, PTP_VENDOR_MICROSOFT },
+       { N_("Fluorescent: Tungsten"),  0x8006, PTP_VENDOR_MICROSOFT },
+       { N_("CT settings"),            0x8007, PTP_VENDOR_MICROSOFT },
+       { N_("Underwater"),             0x8008, PTP_VENDOR_MICROSOFT },
+
 };
 GENERIC16TABLE(WhiteBalance,whitebalance)

gphoto2 examples

camera summary info

gphoto2 --summary
Camera summary:                                                                
Manufacturer: Ricoh Company, Ltd.
Model: RICOH THETA X
  Version: 1.30.0
  Serial Number: 14010001
Vendor Extension ID: 0x6 (1.10)
Vendor Extension Description: 

set to video mode

gphoto2 --set-config=5013=32770

start video

gphoto2 --set-config movie=1

stop video

gphoto2 --set-config=/main/actions/opcode=0x1018,0xFFFFFFFF

start video and stop after specified time

gphoto2 --set-config movie=1 --wait-event=10s --set-config movie=0 

take picture

gphoto2 --trigger-capture

disable image stitching

gphoto2 --set-config /main/other/d834=2

enable image stitching

gphoto2 --set-config /main/other/d834=1

put camera to sleep

gphoto2 --set-config=/main/other/d80e=1

wake camera up from sleep

gphoto2 --set-config=/main/other/d80e=0

get current shutter speed

gphoto2 --get-config shutterspeed
Label: Shutter Speed                                                           
Readonly: 0
Type: RADIO
Current: Auto
Choice: 0 Auto
END

set exposure program to shutter priority

gphoto2 --set-config=/main/other/500e=4

get shutterspeed when camera set to shutter priority

 gphoto2 --get-config shutterspeed
Label: Shutter Speed                                                           
Readonly: 0
Type: RADIO
Current: 1/250
Choice: 0 1/16000
Choice: 1 1/12800
Choice: 2 1/10000
Choice: 3 1/8000
Choice: 4 1/6400
Choice: 5 1/5000
Choice: 6 1/4000
Choice: 7 1/3200
Choice: 8 1/2500
Choice: 9 1/2000
Choice: 10 1/1600
Choice: 11 1/1250
Choice: 12 1/1000
Choice: 13 1/800
Choice: 14 1/640
Choice: 15 1/500
Choice: 16 1/400
Choice: 17 1/320
Choice: 18 1/250
Choice: 19 1/200
Choice: 20 1/160
Choice: 21 1/125
Choice: 22 1/100
Choice: 23 1/80
Choice: 24 1/60
Choice: 25 1/50
Choice: 26 1/40
Choice: 27 1/30
Choice: 28 1/25
Choice: 29 1/20
Choice: 30 1/15
Choice: 31 1/13
Choice: 32 1/10
Choice: 33 1/8
Choice: 34 1/6
Choice: 35 1/5
Choice: 36 1/4
Choice: 37 1/3
Choice: 38 10/25
Choice: 39 1/2
Choice: 40 10/16
Choice: 41 10/13
Choice: 42 1/1
Choice: 43 13/10
Choice: 44 16/10
Choice: 45 2/1
Choice: 46 25/10
Choice: 47 32/10
Choice: 48 4/1
Choice: 49 5/1
Choice: 50 6/1
Choice: 51 8/1
Choice: 52 10/1
Choice: 53 13/1
Choice: 54 15/1
END

set shutterspeed to 1/1600

gphoto2 --set-config-index shutterspeed=10

get shutterspeed to verify setting

Current is 1/1600, which is choice 10.

gphoto2 --get-config shutterspeed
Label: Shutter Speed                                                           
Readonly: 0
Type: RADIO
Current: 1/1600

Set Back to Mode 2, Auto

gphoto2 --set-config=/main/other/500e=2

Reconfirm Auto

gphoto2 --get-config shutterspeed
Label: Shutter Speed                                                           
Readonly: 0
Type: RADIO
Current: Auto
Choice: 0 Auto

get white balance

gphoto2 --get-config /main/imgsettings/whitebalance
Label: WhiteBalance                                                            
Readonly: 0
Type: RADIO
Current: Fluorescent: White
Choice: 0 Automatic
Choice: 1 Daylight
Choice: 2 Shade
Choice: 3 Cloudy
Choice: 4 Tungsten
Choice: 5 Tungsten 2
Choice: 6 Fluorescent: Daylight
Choice: 7 Fluorescent: Day White
Choice: 8 Fluorescent: White
Choice: 9 Fluorescent: Tungsten
Choice: 10 CT settings
Choice: 11 Underwater
END
3 Likes

Lets control that robot and make it dance, please! :slight_smile: work in progress on my end… but I see you are playing around too.

Can we make this robot work through WiFi too? Like UDB cable plugged into NVIDIA Jetson and it connects to a WiFi network?

2 Likes

there are several different possible “WiFi” connection. One is the WiFi connection to control the robot movements. Currently, the robot controller itself has a WiFi connection. The second connection is the camera, either to the camera controller or a router. The connection between the camera and camera controller can either be USB with the USB API or WiFi using the WebAPI.

If the connection between the camera and the camera controller is USB, I can control the camera over WiFi using ssh.

For factory inspection, with the robot connected with WiFi, the operator can turn on the THETA with WiFi from a Windows computer in a remote operating center using ssh command to Jetson using TCP/IP, then the Jetson turns on the Z1 with USB to start broadasting the live stream. The system switches to 360 still image for detailed inspection, then back to live streaming to see a live view of the factory.

This video below shows a mechanical test of mobility, not the full system.

I have more robot videos doing different things. I need to upgrade the power system of the robot as the camera and Jetson are drawing too much power and making the controller unstable.

2 Likes