Saw this video by misoji engineer on using USB live streaming with ROS 3D Robot Visualizer rviz2.
It’s interesting that he cut the 360 image into four segments.
There’s a lot of interesting information on Hackster.
He also has a nice blog in Japanese.
Oh, this is pretty interesting. I’m curious why Rviz2 would divide a 360 image into 4 sections. And other questions. 
I’m going to spend some more time looking into this.
@jcasman please try and contact the author. I don’t believe that he’s on .guide. If you translate his work (he gave us permission), and publish the work, consider asking him to join so we can gave him a community thanks award.
He indicated that he’s aware of .guide and uses the information.
Originally published in Japanese in the blog An Engineer’s Electrician https://misoji-engineer.com/ Full article here.
I will introduce how to install it and provide examples of how to run it.
I used the 360-degree camera RICOH THETA V from the KR260. It is controlled via USB.
Below are some photos taken with a 360° camera.
The actual test video is below.
You can see that it is controlled from the ipynb file.
Next, I will introduce the actual installation procedures and the ipynb file that was used.
The 360° camera used was RICOH’s THETA V.
Various APIs and libraries are available making it an easy-to-use 360° camera.
There are different connection methods such as USB and Bluetooth.
The latest model is the RICOH THETA X, but the reason I chose the V is explained in this article: Reasons for purchasing the THETA V instead of the RICOH THETA X
Connect THETA to the KR260 via USB.
Looking at the dmesg log, I was able to connect without any problems.
ubuntu@kria:~$ sudo su
root@kria:/home/ubuntu# dmesg |grep usb
[ 8.214013] usb 1-1.2: New USB device found, idVendor=05ca, idProduct=0368, bcdDevice= 1.00
[ 8.222380] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 8.229697] usb 1-1.2: Product: RICOH THETA V
[ 8.234055] usb 1-1.2: Manufacturer: Ricoh Company, Ltd.
[ 8.239367] usb 1-1.2: SerialNumber: 00119628
I made and installed it by referring to this GitHub repo: https://github.com/codetricity/libptp2-theta
sudo apt install build-essential
sudo apt install libtool
sudo apt install automake
sudo apt install pkg-config
sudo apt install subversion
sudo apt install libusb-dev
git clone https://github.com/codetricity/libptp2-theta
cd libptp2-theta/
./configure
make
autoreconf -i
./configure
automake
make
sudo make install
sudo ldconfig -v
After installation, you will be able to use the command theta
For example, if you want to check the help, type theta -h
If you want to read the camera (device) information, use theta -i
root@kria:/home/ubuntu# theta -h
root@kria:/home/ubuntu# theta -i
THETA Device Info
==================
Model: RICOH THETA V
manufacturer: Ricoh Company, Ltd.
serial number: '00119628'
device version: 1.00.2
extension ID: 0x00000006
image formats supported: 0x00000004
extension version: 0x006e
Control it from Python using Kria-PYNQ’s Jupyter Notebook.
What I’m introducing here is just one example.
Various other operations are also possible using the official RICOH USB API information: https://github.com/ricohapi/theta-api-specs/tree/main/theta-usb-api
The ipynb file I created and executed this time is below. I have uploaded it to Gist.
Just like with the command, let’s check the camera information using Python.
from subprocess import getoutput
import time
theta_info = getoutput('theta --info')
print(theta_info)
THETA Device Info
==================
Model: RICOH THETA V
manufacturer: Ricoh Company, Ltd.
serial number: '00119628'
device version: 1.00.2
extension ID: 0x00000006
image formats supported: 0x00000004
extension version: 0x006e
I’ll follow the same steps as described on GitHub, but I’m just going to try out the steps up to taking a photo.
First, wake up THETA from power saving mode. The camera’s LED lamp will light up blue.
theta_wakeup = getoutput('theta --set-property=0xD80E --val=0x00')
print(theta_wakeup)
Camera: RICOH THETA V
'UNKNOWN' is set to: 1
Changing property value to 0x00 [(null)] succeeded.
Switch to camera shooting mode. The camera mark will light up blue.
theta_camera_mode = getoutput('theta --set-property=0x5013 --val=0x0001')
print(theta_camera_mode)
Camera: RICOH THETA V
'Still Capture Mode' is set to: [Normal]
Changing property value to 0x0001 [(null)] succeeded.
All you have to do is take a picture (capture).
I checked the shooting time of the 360° camera THETA V, and it was about 3 seconds.
time1 = time.time()
theta_capture = getoutput('theta --capture')
time2 = time.time()
capture_time = time2-time1
print("Performance: {} (s)".format(capture_time))
print(theta_capture)
Performance: 3.1056067943573 (s)
Initiating capture...
Object added 0x000000ee
Capture completed successfully!
The actual conditions around the KR260 and THETA were as follows. (Photo taken with a smartphone.)
You can check the files saved in THETA with theta -L
The data that was obtained is shown in the image below.
theta_list = getoutput('theta -L')
print(theta_list)
Listing files...
Camera: RICOH THETA V
Handler: Size: Captured: name:
0x000000ad: 2042821 2024-03-30 23:35 R0010146.JPG
0x000000ae: 2021885 2024-03-30 23:36 R0010147.JPG
0x000000af: 1801502 2024-03-31 07:01 R0010148.JPG
0x000000b0: 1861412 2024-03-31 08:06 R0010149.JPG
0x000000b1: 1716924 2024-03-31 09:18 R0010150.JPG
0x000000b2: 1723153 2024-03-31 09:22 R0010151.JPG
You can specify files (or all) and save them to the KR260.
theta --get-file=HANDLE
theta --get-all-files
It will be transferred from THETA to the current directory.
theta_get_file = getoutput('theta --get-file=0x00000088')
print(theta_get_file)
Camera: RICOH THETA V
Skipping file: "R0010155.JPG", file exists!
Display the image using OpenCV.
import cv2
from matplotlib import pyplot as plt
image_path = 'R0010109.JPG'
image = cv2.imread(image_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.show()
I was also able to view the 360° camera images taken using OpenCV on ipynb.
I tried operating the 360-degree camera RICOH THETA from a KR260.
However, it takes about three seconds for the camera to take a single picture, so it cannot be used for real-time control as-is.
Therefore, we used a different method to stream the data to OpenCV in real time using Gstreamer.
The tests carried out on the KR260 this time are some of the tests introduced in this article: Impressions from participating in the AMD (Xilinx) FPGA contest
Interesting concept. It would have been good to try this out, but since I don’t have the Theta Camera, does the Theta API work for this?
@shegz101 Yes, it sure does. THETA has multiple APIs. You’ve worked with React Native development and the THETA WebAPI over Wi-Fi. This one is the USB API. It’s used with these kind of configurations, connecting a THETA to a small board computer, and doing live streaming. There are interesting robotics projects, but the USB API is also used in medical devices and for mapping interior spaces in the insurance, residential, and commercial real estate markets. And a lot more.
Added another reply, using the AI Chatbot in the forum:
The RICOH THETA supports multiple APIs to interact with different aspects of the camera. Here are the main APIs available:
Each API serves different needs depending on how you interact with the RICOH THETA camera. For further details, the main documentation is available at RICOH THETA API Documentation.
Awesome. This is impressive! It seems to be just one API I have had to interact with, and I didn’t even know there were more interesting ones. Thank you.