Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

Hello all!

I fixed the issue. It was actually quite simple, I just did not know this was a thing.

Apparently, there is a llinux user group called plugdev that allows access to USB devices from an SSH session. I only had to run this command:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Now everything works fine.

Thanks a lot @craig for your availability, this meant a lot to me.

Take care!

2 Likes

Thank you for posting the solution.

I was not aware of the plugdev group. This will help other Spot users.

have a great holiday seasoon.

tip for gstthetauvc on Jetson Xavier NX

credit alex-uam
reference on github issue

problem

I have compiled the gstthetauvc plugin and am attempting to run it on a Jetson Xavier NX, however I receive the error :

ERROR: from element /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0: Found 1 Theta(s), but none available.

I have successfully gotten it running on a Jetson Xavier AGX, as well as an Ubuntu desktop PC. Just wondering if there are any obvious steps that I’ve missed for the NX? Possibly permission issues?

solution

We managed to find the solution. Turns out it was as simple as setting an environmental variable in bash:

export LD_LIBRARY_PATH=/usr/local/lib

For whatever reason, this is automatically set on Intel based Ubuntu, and the Xavier AGX series board, but not the NX!

:+1::fire:


other

on my (craig) system, this is my /etc/ld.so.conf.d/libc.conf

 cd /etc/ld.so.conf.d/
craig@jetpack-4:/etc/ld.so.conf.d$ ls
aarch64-linux-gnu.conf             fakeroot-aarch64-linux-gnu.conf
aarch64-linux-gnu_EGL.conf         libc.conf
aarch64-linux-gnu_GL.conf          nvidia-tegra.conf
cuda-10-2.conf                     vpi1.conf
fakechroot-aarch64-linux-gnu.conf
craig@jetpack-4:/etc/ld.so.conf.d$ cat libc.conf 
# libc default configuration
/usr/local/lib

Hi all,

I’m working with Theta X and I use gstthetauvc plugin for capturing the video stream. I had some issues using it. Fortunately, I could make it work by using 30 fps in the format settings (instead of 29) (thetauvc_get_stream_ctrl_format_size function in thetauvc.c file).

Now, I created a docker image to ease the deployment of the streaming in several machines. However, I couldn’t make it work. Camera is detected but UVC returns an error of “No such device” when uvc_open() is executed. This is the command that I’m using to create the container:

docker run --network host --privileged --device '/dev/bus/usb' gst-xrlab

And this is the gstreamer pipeline that runs inside the container:

gst-launch-1.0 -v thetauvcsrc ! queue ! h264parse ! decodebin ! queue ! autovideosink sync=false

Does anyone stream using a docker image? Does he/she find this kind of issue? I’d really appreciate any help. Thanks in advance.

KR,

 Juan

did you see the post by @nicodax about the user permissions:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Actually, now that I reread his post, I think he’s using libuvc-theta-sample

Thank you very much for the reply.

I’m afraid that the user that run the container is within plugdev group. Anyway, it’s weird that in my case camera is detected inside the container but it cannot be open. AFAIK, @nicodax had issues detecting the camera.

I unfortunately have not tried the Linux streaming with docker. It seems like you have it working outside of docker, right?

Please post your system config such as Ubuntu 22.04 on x86 or ROS Noetic Ninjemys on Nvidia Xavier NX.

I do not have experience with Docker in general, so the questions below are likely irrelevant as you have streaming working outside of docker.

are /usr/local/lib/ and the location of gstthetauvc.so in your library load path?

You may have done something like this:

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

Note that there is a hyphen between gstreamer and 1.0.

Thanks a lot for the suppport.

The host of the docker image is Ubuntu 20.04 on x86_64 architecture.

In addition, I use the variable GST_PLUGIN_PATH to set where the plugin is located. Plugin is detected and working. This is the log that is generated with gstreamer using: GST_DEBUG=2,thetauvcsrc:6

0:00:00.688153989     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:208:gst_thetauvcsrc_set_property:<thetauvcsrc0> set_property
0:00:00.688169165     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:208:gst_thetauvcsrc_set_property:<thetauvcsrc0> set_property
Setting pipeline to PAUSED ...
0:00:00.689459545     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689466171     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.689476892     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689479785     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.689487561     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689489383     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.689498710     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689500063     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.690952410     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:448:gst_thetauvcsrc_start:<thetauvcsrc0> start
0:00:00.690960576     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:449:gst_thetauvcsrc_start:<thetauvcsrc0> dev=-1 mode=0
0:00:00.760036006     8 0x55dc6da40c00 WARN             thetauvcsrc gstthetauvcsrc.c:493:gst_thetauvcsrc_start:<thetauvcsrc0> error: Found 1 Theta(s), but none available (No such device).
0:00:00.760414751     8 0x55dc6da40c00 WARN                 basesrc gstbasesrc.c:3468:gst_base_src_start:<thetauvcsrc0> error: Failed to start
0:00:00.760480896     8 0x55dc6da40c00 WARN                 basesrc gstbasesrc.c:3824:gst_base_src_activate_push:<thetauvcsrc0> Failed to start in push mode
0:00:00.760501442     8 0x55dc6da40c00 WARN                GST_PADS gstpad.c:1142:gst_pad_set_active:<thetauvcsrc0:src> Failed to activate pad
ERROR: Pipeline doesn't want to pause.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0: Found 1 Theta(s), but none available (No such device).
Additional debug info:
gstthetauvcsrc.c(493): gst_thetauvcsrc_start (): /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0
Freeing pipeline ...
0:00:00.761629738     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:270:gst_thetauvcsrc_finalize:<thetauvcsrc0> finalize

KR,

Juan

Hi again,

I tested a Jetson Nano having exactly the same issue that I got using docker. Some remarks that might be important:

  • I got the same error using gstthetauvc plugin and libuvc-theta-sample: libuvc raised an error of “Not supported (code -12)” when uvc_open() ran.
  • Same code and procedures worked fine in a laptop running Ubuntu 20.04.
  • My camera is Theta X. From this thread, it seems that the same tools works without issues with Z1 and V.
  • Same issue running as root. Just in case, I also added the login user to plugdev group (sudo usermod -aG plugdev USER).
  • Camera was detected by uvc (of course, using lsusb and usb-devices as well).

I’d really appreciate any help. Thanks in advance!

Juan

sorry that I can’t be more helpful, but I’m not able to figure out the source of the problem. Maybe other people can suggest a solution.

In the version of gstthetauvc you are using, is the USBPID_THETAX_UVC defined as 0x2717 in thetauvc.h?

I can try and test the X with my Jetson Nano.

Test on Jetson Nano, March 6, 2023.

both gstthetauvc and libuvc-theta-sample work with RICOH THETA X

THETA Device Info
==================
Model: RICOH THETA X
  manufacturer: Ricoh Company, Ltd.
  serial number: '14010001'
  device version: 1.41.0
  extension ID: 0x00000006
  image formats supported: 0x00000004
  extension version: 0x006e

Make sure the camera is not mounted by the OS.

Jetpack 4.6

cat /etc/nv_tegra_release 
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021

or using apt-cache

apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.6-b199
Architecture: arm64
Maintainer: NVIDIA Corporation

opencv test

cat > sample.py
import cv2
# pipeline below worked on Jetson and x86
cap = cv2.VideoCapture("thetauvcsrc \
    ! decodebin \
    ! autovideoconvert \
    ! video/x-raw,format=BGRx \
    ! queue ! videoconvert \
    ! video/x-raw,format=BGR ! queue ! appsink")

# pipeline suggestion thanks to nickel110
# attempt to force hardware acceleration
# tested with NVIDIA 510.73 with old GTX 950 on Ubuntu 22.04
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! queue \
#     ! h264parse \
#     ! nvdec \
#     ! gldownload \
#     ! queue \
#     ! videoconvert n-threads=0 \
#     ! video/x-raw,format=BGR \
#     ! queue \
#     ! appsink")

# NVIDIA Jetson
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! nvv4l2decoder \
#     ! nvvidconv \
#     ! video/x-raw,format=BGRx \
#     ! queue ! videoconvert \
#     ! video/x-raw,format=BGR ! queue ! appsink")

if not cap.isOpened():
    raise IOError('Cannot open RICOH THETA')

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

run it

python sample.py 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=1, duration=-1

full build of gstthetauvc, installation and test

using RICOH THETA X

make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
craig@jetpack-4:~/Development/gstthetauvc/thetauvc$ ls
gstglutils.c         gstthetauvc.c     Makefile    thetauvc.h
gstglutils.h         gstthetauvc.so    obj
gstthetatransform.c  gstthetauvcsrc.c  shader.h
gstthetatransform.h  gstthetauvcsrc.h  thetauvc.c
craig@jetpack-4:~/Development/gstthetauvc/thetauvc$ sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
craig@jetpack-4:~/Development/gstthetauvc/thetauvc$ gst-launch-1.0 thetauvcsrc mode=4K ! queue ! h264parse ! nvv4l2decoder ! queue ! nv3dsink sync=false
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
XIO:  fatal IO error 0 (Success) on X server ":0"
      after 844 requests (844 known processed) with 0 events remaining.

action shots

  • THETA X firmware 1.41
  • gstthetauvc
  • NVIDIA Jetson Nano running Jetpack 4.6

gstthetauvc update on March 4, 2023

framerate negotiation updated.

1 Like

Thank you for your test. Great news! So, I know that I must be able to make it work now. Although I haven’t done yet. Theta X pid is defined in my source code. If it wasn’t, camera wouldn’t be detected.

But, I don’t understand what you said:

Camera is already mounted by the OS. At least, I only plug the camera and it appears as USB device. I don’t know how to do it in another way.

user@jetson:~$ lsusb
Bus 001 Device 003: ID 05ca:0373 Ricoh Co., Ltd 
user@jetson:~$ ls -l /dev/bus/usb/001/003 
crw-rw-r--+ 1 root plugdev 189, 2 mar  6 17:19 /dev/bus/usb/001/003

KR,

  Juan

If the camera is mounted as a filesystem, the stream may not work on the NVIDIA Jetson devices. See the following article.

this older article may also be relevant.

you can try kill the gphoto2-volume-monitor and spawner

hi, @craig ,
it may not be relevant to this, but while I was doing in past experiments with USB connected Theta Z1, I noticed some difference when on Z1, the “Live” mode was turned on or off. I do not remember exactly, but I was having issues at that time, it may or may not be there at Theta X. But also how device was detected by my Raspberry Pi 4 running ubuntu, it was different depending on actual stat of camera (MODE button on Z1).

The USB ID is going to be different if the camera is in still image mode. However, I think that @buburider has the camera in live streaming mode.

#define USBPID_THETAX_UVC 0x2717

Example When X Camera in Still Image mode

Device ID is 2718.

craig@craig-desktop:~$ lsusb
...
Bus 005 Device 002: ID 05ca:2718 Ricoh Co., Ltd RICOH THETA X
...

Example when in live streaming mode

device id is 2717

craig@craig-desktop:~$ lsusb
Bus 005 Device 003: ID 05ca:2717 Ricoh Co., Ltd RICOH THETA X

default behavior is to mount THETA X as a storage device

image

You must unmount it

As @craig said I’m using live streaming mode. So the USB ID is 0x2717.

I’ve just found the issue: my jetson computer had a previous libuvc library installed. So, instead of using libuvc-theta, the application loaded the native version. This is the reason why the same code was working in my laptop. In the laptop, the native version of libuvc didn’t exist. After resolving that, it works fine.

Thank you very much for the help. It’s really awesome to have this forum to share results and issues.

KR,

  Juan
2 Likes

First, congratulations on getting it working!

Thank you for sharing the solution.

I’m sorry that I didn’t recommend this earlier. This is a known problem and I was just researching the issue last week when you asked the question. However, I didn’t make the connection between the previous libuvc and your environment. I’m sharing the additional information below to help other people with this problem. Thanks again for contributing your solution.

There is a note from the author of gstthetauvc

image

this was the original problem.

1 Like

great to hear. If you ever talk to anyone at RICOH, please mention that this forum was useful. We’re dependent on continued sponsorship from RICOH to keep the independent forum going. :slight_smile:

Also, I’ve just updated this document with the error message first reported and your solution.

Help - RICOH THETA Development on Linux

very glad that you got it working. I respect that you kept going even when the source of the problem was tough to trace.

1 Like

I’ve been using Jetpack 4.6, which corresponds to L4T R32, Revision 6.1.

cat /etc/nv_tegra_release 
# R32 (release), REVISION: 6.1

I recently met with a community member running L4T R32, Revision 7.1 and having problems with RICOH THETA X firmware 2.10.1.

To replicate the test with the newer version of Jetpack, I’m going first take a baseline test with Jetpack 4.6. I’ll then update this post with the install of Jetpack 4.6.4 (newest for Nano).

In this example, I’ve renamed ptpcam to theta in order to test the modified and unmodified versions of ptpcam on the same Jetson.

theta --info

THETA Device Info
==================
Model: RICOH THETA X
  manufacturer: Ricoh Company, Ltd.
  serial number: '14010001'
  device version: 2.10.1
  extension ID: 0x00000006
  image formats supported: 0x00000004
  extension version: 0x006e
ls
gst_loopback  gst_viewer.c  Makefile    thetauvc.h
gst_viewer    gst_viewer.o  thetauvc.c  thetauvc.o
craig@jetpack-4:~/Development/libuvc-theta-sample/gst$ ./gst_loopback 
start, hit any key to stop
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 

latency

At 4K, latency is 630ms.

I believe that the top/bottom correction of the live stream can be disabled with this api

theta-api-specs/theta-web-api-v2.1/options/_top_bottom_correction.md at main · ricohapi/theta-api-specs · GitHub

This would likely reduce latency.

install jetpack

Using etcher with this file.

image

There appears to be another install method using the SDK manager.

SDK Manager | NVIDIA Developer

I decided to use etcher.

after upgrade

R32 release is now higher. Previous was 6.1.

cat /etc/nv_tegra_release 
# R32 (release), REVISION: 7.1

/usr/local/lib seems like it will load.

/etc/ld.so.conf.d$ cat libc.conf 
# libc default configuration
/usr/local/lib

from home directory

mkdir Development
cd Development/
git clone https://github.com/ricohapi/libuvc-theta.git 
cd libuvc-theta/
mkdir build
cd build/
cmake ..
-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR) 
-- Checking for module 'libjpeg'
--   No package 'libjpeg' found
-- Looking for pthread_create in pthreads - not found

sudo apt install libjpeg-dev
sudo apt install doxygen

rerun cmake …

cmake ..
-- libusb-1.0 found using pkgconfig
-- Found JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so  
-- Found JPEG library using standard module
-- Building libuvc with JPEG support.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/craig/Development/libuvc-theta/build
make
sudo make install
[ 45%] Built target uvc_static
[ 90%] Built target uvc
[100%] Built target example
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libuvc.so.0.0.6
-- Installing: /usr/local/lib/libuvc.so.0
-- Installing: /usr/local/lib/libuvc.so
-- Installing: /usr/local/include/libuvc/libuvc.h
-- Installing: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/libuvc.a
-- Up-to-date: /usr/local/include/libuvc/libuvc.h
-- Up-to-date: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets-release.cmake
-- Installing: /usr/local/lib/cmake/libuvc/FindLibUSB.cmake
-- Installing: /usr/local/lib/cmake/libuvc/FindJpegPkg.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfigVersion.cmake
-- Installing: /usr/local/lib/pkgconfig/libuvc.pc
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfig.cmake

use ldconfig

ldconfig -v
/usr/local/lib:
	libuvc.so.0 -> libuvc.so.0.0.6

install libuvc-theta-sample

modified version

git clone https://github.com/codetricity/libuvc-theta-sample.git

overkill. install everything related to gstreamer.

sudo apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio libgstreamer-plugins-base1.0-dev

modify gst_viewer.c for single camera on nano.

if (strcmp(cmd_name, "gst_loopback") == 0)
        pipe_proc = "decodebin ! autovideoconvert ! "
                "video/x-raw,format=I420 ! identity drop-allocation=true !"
                "v4l2sink device=/dev/video0 qos=false sync=false";

make

cd gst/
make

error

./gst_viewer 
works as expected

install gstthetauvc

git clone https://github.com/nickel110/gstthetauvc
Cloning into 'gstthetauvc'...
cd gstthetauvc/thetauvc
make
sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

gstthetauvc test

gst-launch-1.0 thetauvcsrc mode=4K ! queue ! h264parse ! nvv4l2decoder ! queue ! nv3dsink sync=false
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

opencv test

pip install opencv-python

Found 1 Theta(s), but none available.


I received a question from a community member about the error message above. There is information in the post below and especially the issue on GitHub.

Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson - #348 by craig.


The main point is to purge libuvc.so from your system, which may be in libuvc-dev.

sudo apt purge libuvc-dev