Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

@iketani_takuya

Can you post debugging information for the gstreamer pipeline using gstthetauvc

  1.  Set GST_DEBUG environment variable to output debug information.
    

Debug level can be specified per plugin basis, please refer to
Basic tutorial 11: Debugging tools
If you want to suppress color output, set GST_DEBUG_NO_COLOR=1 as well.

  1.  Run the pipeline partially.
    

With fakesink plugin, you can run partial pipeline.
E.g.

gst-launch-1.0 thetauvcsrc num-buffers=150 ! queue ! h264parse ! fakesink

Adding plugin one by one, you can find plugin which makes error.


I got a tip from the nickel110 that you can confirm gstreamer support in cv2 with the following:

python3
Python 3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.videoio_registry.hasBackend(cv2.CAP_GSTREAMER)
True
>>> 

@dvzt
This works

import cv2
# pipeline below worked
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! decodebin \
#     ! autovideoconvert \
#     ! video/x-raw,format=BGRx \
#     ! queue ! videoconvert \
#     ! video/x-raw,format=BGR ! queue ! appsink")

# pipeline suggestion thanks to nickel110
# attempt to force hardware acceleration
# tested with NVIDIA 510.73 with old GTX 950 on Ubuntu 22.04
cap = cv2.VideoCapture("thetauvcsrc \
    ! queue \
    ! h264parse \
    ! nvdec \
    ! gldownload \
    ! queue \
    ! videoconvert n-threads=0 \
    ! video/x-raw,format=BGR \
    ! queue \
    ! appsink")

if not cap.isOpened():
    raise IOError('Cannot open RICOH THETA')

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

Approximately 450ms latency when used with Python script above and stream processed with OpenCV per the cv2.resize above.

Canny Edge Detection


Gray scale, Gassian Blur, and Canny Edge simultaneously.

It looks like nickel110 added THETA X PID detection to gstthetauvc. I’ll give this a try soon. I’ve only used gstthetauvc with the THETA Z1 and THETA V. Both worked great and seem like a better way to connect than with v4l2loopback due to the simplicity of using gstthetauvc compared to the combination of libuvc-theta-sample and libuvc-theta and v4l2loopback

@craig Its working well, thank you for the example, and the performance is quiet good

2 Likes

Nice! Great to hear. :slight_smile:

Is the performance noticeably better with nvdec and glimagedownload?

You have a much faster computer and GPU than I do. Are you using CUDA with OpenCV? I was doing some work with CUDA a while ago, but stopped as it was a little difficult for me, but it was faster.

1 Like

gstthetauvc on Jetson Nano

make

cd thetauvc/
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ ls
gstglutils.c         gstthetatransform.h  gstthetauvcsrc.h  thetauvc.c
gstglutils.h         gstthetauvc.c        Makefile          thetauvc.h
gstthetatransform.c  gstthetauvcsrc.c     shader.h
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ 

install

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

use

gst-launch-1.0 thetauvcsrc mode=4K ! queue ! h264parse ! nvv4l2decoder ! queue ! nv3dsink sync=false

result

Successful

OpenCV Test

import cv2
# pipeline below worked on Jetson and x86
cap = cv2.VideoCapture("thetauvcsrc \
    ! decodebin \
    ! autovideoconvert \
    ! video/x-raw,format=BGRx \
    ! queue ! videoconvert \
    ! video/x-raw,format=BGR ! queue ! appsink")

# pipeline suggestion thanks to nickel110
# attempt to force hardware acceleration
# tested with NVIDIA 510.73 with old GTX 950 on Ubuntu 22.04
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! queue \
#     ! h264parse \
#     ! nvdec \
#     ! gldownload \
#     ! queue \
#     ! videoconvert n-threads=0 \
#     ! video/x-raw,format=BGR \
#     ! queue \
#     ! appsink")

# NVIDIA Jetson
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! nvv4l2decoder \
#     ! nvvidconv \
#     ! video/x-raw,format=BGRx \
#     ! queue ! videoconvert \
#     ! video/x-raw,format=BGR ! queue ! appsink")

if not cap.isOpened():
    raise IOError('Cannot open RICOH THETA')

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

image


Find JetPack version - Method #1

sudo apt-cache show nvidia-jetpack
[sudo] password for craig: 
Package: nvidia-jetpack
Version: 4.4.1-b50
Architecture: arm64
Maintainer: NVIDIA Corporation

Find JetPack Version - Method #2

cat /etc/nv_tegra_release 
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t210ref, EABI: aarch64, DATE: Fri Oct 16 19:44:43 UTC 2020

@craig
We apologize for our late reply, and we would like to report our current situation.
Although we downgrade our Jetpack from 4.5 to 4.3, the same error still appeared.
(gst_viewer worked correctly, but gst_loopback didn’t work)
We ran the command “gst-launch-1.0 thetauvcsrc num-buffers=150 ! queue ! h264parse ! fakesink”, it showed waring with “erroneous pipeline: no element thetauvcsrc”.
Since we haven’t read Basic tutorial thoroughly, we would like to debug the plugins later.

Did you install additional software after you installed JetPack? Do you have another version of libuvc installed?

Do you have the gstthetauvc.so installed in gstreamer-1.0 and not in gstreamer1.0?

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

What is the output of make?

cd thetauvc/
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ ls
gstglutils.c         gstthetatransform.h  gstthetauvcsrc.h  thetauvc.c
gstglutils.h         gstthetauvc.c        Makefile          thetauvc.h
gstthetatransform.c  gstthetauvcsrc.c     shader.h
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ 

@iketani_takuya
I have streaming on my Jetson Nano working now with Jetpack 4.6.

Try uninstalling docker and make sure the camera is not mounted. Maybe something is interfering with the USB connection to the camera?


mkdir build
cd build/
cmake ..
sudo make install
...
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libuvc.so.0.0.6
-- Installing: /usr/local/lib/libuvc.so.0
-- Installing: /usr/local/lib/libuvc.so
-- Installing: /usr/local/include/libuvc/libuvc.h
-- Installing: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/libuvc.a
-- Up-to-date: /usr/local/include/libuvc/libuvc.h
-- Up-to-date: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets-release.cmake
-- Installing: /usr/local/lib/cmake/libuvc/FindLibUSB.cmake
-- Installing: /usr/local/lib/cmake/libuvc/FindJpegPkg.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfigVersion.cmake
-- Installing: /usr/local/lib/pkgconfig/libuvc.pc
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfig.cmake

...
sudo ldconfig

sudo modprobe v4l2loopback
lsmod |grep v4l2
v4l2loopback           37127  0
 
ls -l /sys/devices/virtual/video4linux/
total 0
drwxr-xr-x 3 root root 0 Jul  6 15:18 video0

./gst_loopback 
start, hit any key to stop
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 


sudo apt install vlc

sudo apt-cache show nvidia-jetpack

[sudo] password for craig:

Package: nvidia-jetpack

Version: 4.6-b199

Architecture: arm64

Maintainer: NVIDIA Corporation

Installed-Size: 194

cat /etc/nv_tegra_release

# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021

works on /dev/video0 with vlc

Z1 is unmounted

gstthetauvc

did you do this?

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

and then restart the Jetson?

Hello all,

I am trying to develop a service for accessing cameras emitting from a Spot robot.

I plugged an RICOH THETA Z1 360 usb camera to the robot’s core and am trying to map the device to /dev/video0 (there are no other cameras on the core).

I followed this tutorial for setting up libuvc-theta and libuvc-theta-sample on the robot’s core.

I also followed this OpenCV Python demo and installed v4l2loopback on the robot.

When I try to run ./gst_loopback with the camera connected and in live stream mode, I get a Can't open THETA error.

Before testing this process on the robot’s core, I tried it on my local dev machine (ubuntu) and everything worked fine.

I am not very comfortable reading c code and i was not able to debug it myself…
It seems to be coming from the line 238 of the gst_viewer.c file:

res = uvc_open(dev, &devh);

Can someone explain what this does and how to output error logs from this command ?

Thanks in advance,

nicodax

PS: i edited the line 190 according to what i read in this forum’s discussion (qos=false and /dev/video0)

I do not have access to a spot. Is it a form of Linux?

consider gstthetauvc

Assuming you are using OpenCV, there are two ways to access the camera:

  1. v4l2loopback will create a virtual video device /dev/video0 or /dev/video1 that you can use to access the device from OpenCV
  2. use gstthetauvc to eliminate v4l2loopback

See this article on using gstthetauvc

Software - RICOH THETA Development on Linux

gstthetauvc is available here:

GitHub - nickel110/gstthetauvc: Gstreamer theta uvc plugin

The recommended approach is to use gstthetauvc instead of v4l2loopback.

The tutorial you are looking at is a bit old and most of the content was written before gstthetauvc was released. I should update it, but haven’t had the time. If there’s more interest in this type of deployment, I will prioritize the documentation update. If you are able, can you tell me what you are using the spot for and if it is a large deployment. This forum receives sponsorship from RICOH and I like to tell them that our documentation is in response to community demand. You can also send me a direct message if your deployment information is not appropriate to post in a public forum.

Can't open THETA error

This indicates that the device cannot be found on the spot. Does the spot already have another video camera at /dev/video0? If not, change line 190 to /dev/video0.

on the spot, can you see the THETA Z1 with gst_viewer ? Not gst_loopback.

$ ./gst_viewer -l
No : Product            : Serial    
 0 : RICOH THETA Z1     : 10010104  

on the spot, can you see the v4l2loopback module loaded in the kernel?

$ lsmod
Module                  Size  Used by
uvcvideo               88565  0
bnep                   16562  2
zram                   26166  4
overlay                48691  0
spidev                 13282  0
v4l2loopback           37383  0

or

on the spot, can you see the dummy video device.

$ v4l2-ctl --info
Driver Info (not using libv4l2):
	Driver name   : v4l2 loopback
	Card type     : Dummy video device (0x0000)
	Bus info      : platform:v4l2loopback-000
	Driver version: 4.9.140
	Capabilities  : 0x85208003

On the spot, can you see the video device with lsusb

image

Feel free to post more questions.

We will get you up and running!

Hello again!

Yes, I’m sorry I wasn’t thorough enough while describing my issue. Spot is a robot manufactured by Boston Dynamics and Spot’s CORE is an onboard computer running on Ubuntu 18.04 on the robot I am working with (not sure if this is a standard).

Thank you for the links about gstthetauvc, I will check it right after this message.

In a nutshell, the Spot robot I work with is owned by a research institute in the construction field. Right now, I am trying to build a web interface that connects to the robot and fetches the live video feeds from all its cameras (built-in or otherwise). Fetching the built-in ones was not a problem. Fetching this 360 RICOH USB cam is proving a bit more challenging. I only need it to be available as a /dev/video* device so that I can reuse my existing and working python scripts allowing the communication between the robot and another machine on the same network. I don’t think I have to go into more details about these scripts but if you are interested, I based my development off of this official demo.

About the Can’t open THETA error:

  • Indeed, ./gst_viewer -l does not see the RICOH THETA Z1 camera:
spot@SpotCORE:~/src/libuvc-theta-sample/gst$ ./gst_viewer -l
No : Product            : Serial    
 0 : (null)             : (null)
  • I can however see the v4l2loopback module loaded in the kernel (provided I load it after booting with the modprobe v4l2loopback command - I did not set it up to automatically load on boot yet);
  • I can also see the dummy video device:
spot@SpotCORE:~/src/libuvc-theta-sample/gst$ sudo v4l2-ctl --info
Driver Info (not using libv4l2):
	Driver name   : v4l2 loopback
	Card type     : Dummy video device (0x0000)
	Bus info      : platform:v4l2loopback-000
	Driver version: 4.15.18
	Capabilities  : 0x85208003

...

  • similarly, I can see the device with lsusb:
# The camera is in livestream mode:
spot@SpotCORE:~/src/libuvc-theta-sample/gst$ lsusb
Bus 002 Device 003: ID 05ca:2715 Ricoh Co., Ltd 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
# The camera went back to sleep:
spot@SpotCORE:~/src/libuvc-theta-sample/gst$ lsusb
Bus 002 Device 004: ID 05ca:036d Ricoh Co., Ltd 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Thanks a lot for the swift response sir, I was not expecting such availability.

Tell me if you are looking for a more specific description of what it is I am trying to do.

Good day to you,

nicodax

Sorry for the second message, I just started reading the links you sent and I realised I was not clear enough (again) on my description.

The scripts I mentioned above are packaged into what Boston Dynamics calls an extension. Essentially it is a Docker container running a service.

For making the video devices available to the container, I have to run it with the --device argument.

As it stands, but I might have to change all of this, I need the device to be available as a video device otherwise I don’t know how to access the video feed.

I’ll keep reading the links you provided of course but I have a feeling this is not the solution I am looking for.

Thanks again,

nicodax

Hello again,

I have an update on the situation.

The Spot CORE has an output HDMI port and I used it to connect an external monitor for testing purposes and shazam! it does work this way. Is there any reason for this ?

I checked again and it does not work when the monitor is plugged out.

I read somewhere that Xorg would not start unless a monitor was plugged in, so I tried starting it manually to see if that would solve the issue, but it doesn’t.

Maybe someone with better knowledge would know what it means ?

I juste checked the thetauvc.c file and indeed both lines you pointed out were missing. I added them and ran make clean && make to rebuild it but it doesn’t seem to have done anything.

nicodax

sorry about my earlier post. I got confused for a moment and thought you were using the THETA X instead of the THETA Z1. I deleted my earlier post.

I don’t have a good solution right now.

I don’t know about X.org monitor issues. Hopefully, someone else will be able to comment on this. You could also try an X.org-specific group on Reddit or stackoverflow. It seems like the core problem is using v4l2loopback with X.org without a monitor.

If it works with the monitor, maybe try a dummy HDMI plug as part of the debugging process and see if it works without a monitor and just the dummy plug?

Are you unable to use gstthetauvc ?

Does docker prevent you from using libusb, libuvc directly?


I do not use docker. I just searched for ideas about how to access USB devices on Docker and this post came up.


I’m reaching the limits of the knowledge that I can offer. However, feel free to keep posting as there are other people more knowledgeable than I am. It may take some time to get an answer. Just letting you know that my ideas may waste your time, so choose the best course.

Hello all!

I fixed the issue. It was actually quite simple, I just did not know this was a thing.

Apparently, there is a llinux user group called plugdev that allows access to USB devices from an SSH session. I only had to run this command:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Now everything works fine.

Thanks a lot @craig for your availability, this meant a lot to me.

Take care!

2 Likes

Thank you for posting the solution.

I was not aware of the plugdev group. This will help other Spot users.

have a great holiday seasoon.

tip for gstthetauvc on Jetson Xavier NX

credit alex-uam
reference on github issue

problem

I have compiled the gstthetauvc plugin and am attempting to run it on a Jetson Xavier NX, however I receive the error :

ERROR: from element /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0: Found 1 Theta(s), but none available.

I have successfully gotten it running on a Jetson Xavier AGX, as well as an Ubuntu desktop PC. Just wondering if there are any obvious steps that I’ve missed for the NX? Possibly permission issues?

solution

We managed to find the solution. Turns out it was as simple as setting an environmental variable in bash:

export LD_LIBRARY_PATH=/usr/local/lib

For whatever reason, this is automatically set on Intel based Ubuntu, and the Xavier AGX series board, but not the NX!

:+1::fire:


other

on my (craig) system, this is my /etc/ld.so.conf.d/libc.conf

 cd /etc/ld.so.conf.d/
craig@jetpack-4:/etc/ld.so.conf.d$ ls
aarch64-linux-gnu.conf             fakeroot-aarch64-linux-gnu.conf
aarch64-linux-gnu_EGL.conf         libc.conf
aarch64-linux-gnu_GL.conf          nvidia-tegra.conf
cuda-10-2.conf                     vpi1.conf
fakechroot-aarch64-linux-gnu.conf
craig@jetpack-4:/etc/ld.so.conf.d$ cat libc.conf 
# libc default configuration
/usr/local/lib

Hi all,

I’m working with Theta X and I use gstthetauvc plugin for capturing the video stream. I had some issues using it. Fortunately, I could make it work by using 30 fps in the format settings (instead of 29) (thetauvc_get_stream_ctrl_format_size function in thetauvc.c file).

Now, I created a docker image to ease the deployment of the streaming in several machines. However, I couldn’t make it work. Camera is detected but UVC returns an error of “No such device” when uvc_open() is executed. This is the command that I’m using to create the container:

docker run --network host --privileged --device '/dev/bus/usb' gst-xrlab

And this is the gstreamer pipeline that runs inside the container:

gst-launch-1.0 -v thetauvcsrc ! queue ! h264parse ! decodebin ! queue ! autovideosink sync=false

Does anyone stream using a docker image? Does he/she find this kind of issue? I’d really appreciate any help. Thanks in advance.

KR,

 Juan

did you see the post by @nicodax about the user permissions:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Actually, now that I reread his post, I think he’s using libuvc-theta-sample