Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

craig · May 31, 2022, 11:14pm

Oh, you have everything working! nice. You’re just trying to get more out of the hardware acceleration.

Does the following work, but you can’t use nvdec and glimagesink with OpenCV VideoCapture cap?

gst-launch-1.0 thetauvcsrc mode=4K ! queue ! h264parse ! nvdec ! queue ! glimagesink sync=false

I have not tried the following:

VideoCapture cap("thetauvcsrc ! nvdec ! autovideoconvert ! video/x-raw,format=BGRx ! queue ! videoconvert ! video/x-raw,format=BGR ! queue ! glimagesink");

Did you have to recompile gstreamer with OpenCV support? Or, was it already compiled in from your Linux distro (like Ubuntu 20.04)?

Maybe this package is enough? libgstreamer-opencv1.0-0

dvzt · June 1, 2022, 7:30am

@craig Yes, I want to get out more from the hardware acceleration.
I built the opencv with gstreamer support and nickel110’s OpenCV example works fine, so I think that’s not the problem. (only the nvdec pipeline crashes)
I tried what you posted

VideoCapture cap("thetauvcsrc ! nvdec ! autovideoconvert ! video/x-raw,format=BGRx ! queue ! videoconvert ! video/x-raw,format=BGR ! queue ! glimagesink");

but I get this:

E0601 09:44:33.237895273    9936 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
[ WARN:0@9.437] global /home/david/develop/opencv/modules/videoio/src/cap_gstreamer.cpp (1226) open OpenCV | GStreamer warning: cannot find appsink in manual pipeline
[ WARN:0@9.437] global /home/david/develop/opencv/modules/videoio/src/cap_gstreamer.cpp (862) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ERROR:0@9.442] global /home/david/develop/opencv/modules/videoio/src/cap.cpp (164) open VIDEOIO(CV_IMAGES): raised OpenCV exception:

OpenCV(4.5.5-dev) /home/david/develop/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): thetauvcsrc mode=4K ! nvdec ! autovideoconvert ! video/x-raw,format=BGRx ! queue ! videoconvert ! video/x-raw,format=BGR ! queue ! glimagesink in function 'icvExtractPattern'

craig · June 1, 2022, 12:56pm

Did you try it with the appsink in the pipeline instead of glimagesink? Just using the nvdec?

dvzt · June 1, 2022, 1:08pm

That was my first idea, but I got this:

OpenCV(4.5.5-dev) /home/david/develop/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): thetauvcsrc mode=4K ! nvdec ! autovideoconvert ! video/x-raw,format=BGRx ! queue ! videoconvert ! video/x-raw,format=BGR ! queue ! appsink in function 'icvExtractPattern'

craig · June 1, 2022, 5:14pm

dvzt:

OpenCV(4.5.5-dev) /home/david/develop/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): thetauvcsrc mode=4K ! nvdec ! autovideoconvert ! video/x-raw,format=BGRx ! queue ! videoconvert ! video/x-raw,format=BGR ! queue ! appsink in function 'icvExtractPattern'

Thanks for posting these detailed reports. Unfortunately, I’m a bit stuck right now, but I hope that other people may be able to help. Just so we have a good record of what works and where people ahve problems, can you list your video GPU model and CPU.

At least you have the THETA working with OpenCV. That must be satisfying.

dvzt · June 1, 2022, 6:29pm

@craig
OS: Ubuntu 22.04 LTS
GPU: Nvidia GTX 1650
CPU: AMD® Ryzen 7 4800h

craig · June 3, 2022, 3:01pm

Maybe you are the same guy davzoltan on GitHub, right? This is the response from the developer

As I don't have discrete GPU PC, I don't have any pipeline example using nvdec.
But according to https://codetricity.github.io/theta-linux/optimization/#configuration-with-v4l2loopack-on-devvideo ,
I guess
`thetauvcsrc ! queue ! h264parse ! nvdec ! gldownload ! queue ! videoconvert n-threads=0 ! video/x-raw,format=BGR ! queue ! appsink`
may work.
Of course, above example isn't optimal, so additional tuning is needed for better performance.

reference

nvdec, gldownload, glimagesink works with thetauvcsrc

gst-launch-1.0 thetauvcsrc ! queue ! h264parse ! nvdec ! gldownload ! queue ! videoconvert n-threads=0 ! video/x-raw,format=BGR ! queue ! glimagesink

craig · June 3, 2022, 4:09pm

@iketani_takuya

Can you post debugging information for the gstreamer pipeline using gstthetauvc

 Set GST_DEBUG environment variable to output debug information.

Debug level can be specified per plugin basis, please refer to
Basic tutorial 11: Debugging tools
If you want to suppress color output, set GST_DEBUG_NO_COLOR=1 as well.

```
 Run the pipeline partially.
```

With fakesink plugin, you can run partial pipeline.
E.g.

gst-launch-1.0 thetauvcsrc num-buffers=150 ! queue ! h264parse ! fakesink

Adding plugin one by one, you can find plugin which makes error.

I got a tip from the nickel110 that you can confirm gstreamer support in cv2 with the following:

python3
Python 3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.videoio_registry.hasBackend(cv2.CAP_GSTREAMER)
True
>>>

craig · June 3, 2022, 8:59pm

dvzt:

But when I try to use the provided gst-launch parameters
thetauvcsrc mode=4K ! queue ! h264parse ! nvdec ! queue ! glimagesink sync=false 
in the OpenCV, it crashes.
Do you have some example for how to Read into OpenCV ?

@dvzt
This works

import cv2
# pipeline below worked
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! decodebin \
#     ! autovideoconvert \
#     ! video/x-raw,format=BGRx \
#     ! queue ! videoconvert \
#     ! video/x-raw,format=BGR ! queue ! appsink")

# pipeline suggestion thanks to nickel110
# attempt to force hardware acceleration
# tested with NVIDIA 510.73 with old GTX 950 on Ubuntu 22.04
cap = cv2.VideoCapture("thetauvcsrc \
    ! queue \
    ! h264parse \
    ! nvdec \
    ! gldownload \
    ! queue \
    ! videoconvert n-threads=0 \
    ! video/x-raw,format=BGR \
    ! queue \
    ! appsink")

if not cap.isOpened():
    raise IOError('Cannot open RICOH THETA')

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

Approximately 450ms latency when used with Python script above and stream processed with OpenCV per the cv2.resize above.

Canny Edge Detection

Gray scale, Gassian Blur, and Canny Edge simultaneously.

craig · June 6, 2022, 5:19pm

It looks like nickel110 added THETA X PID detection to gstthetauvc. I’ll give this a try soon. I’ve only used gstthetauvc with the THETA Z1 and THETA V. Both worked great and seem like a better way to connect than with v4l2loopback due to the simplicity of using gstthetauvc compared to the combination of libuvc-theta-sample and libuvc-theta and v4l2loopback

dvzt · June 7, 2022, 7:21am

@craig Its working well, thank you for the example, and the performance is quiet good

craig · June 7, 2022, 10:56am

Nice! Great to hear.

Is the performance noticeably better with nvdec and glimagedownload?

You have a much faster computer and GPU than I do. Are you using CUDA with OpenCV? I was doing some work with CUDA a while ago, but stopped as it was a little difficult for me, but it was faster.

craig · June 14, 2022, 11:30pm

gstthetauvc on Jetson Nano

make

cd thetauvc/
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ ls
gstglutils.c         gstthetatransform.h  gstthetauvcsrc.h  thetauvc.c
gstglutils.h         gstthetauvc.c        Makefile          thetauvc.h
gstthetatransform.c  gstthetauvcsrc.c     shader.h
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$

install

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

use

gst-launch-1.0 thetauvcsrc mode=4K ! queue ! h264parse ! nvv4l2decoder ! queue ! nv3dsink sync=false

result

Successful

OpenCV Test

import cv2
# pipeline below worked on Jetson and x86
cap = cv2.VideoCapture("thetauvcsrc \
    ! decodebin \
    ! autovideoconvert \
    ! video/x-raw,format=BGRx \
    ! queue ! videoconvert \
    ! video/x-raw,format=BGR ! queue ! appsink")

# pipeline suggestion thanks to nickel110
# attempt to force hardware acceleration
# tested with NVIDIA 510.73 with old GTX 950 on Ubuntu 22.04
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! queue \
#     ! h264parse \
#     ! nvdec \
#     ! gldownload \
#     ! queue \
#     ! videoconvert n-threads=0 \
#     ! video/x-raw,format=BGR \
#     ! queue \
#     ! appsink")

# NVIDIA Jetson
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! nvv4l2decoder \
#     ! nvvidconv \
#     ! video/x-raw,format=BGRx \
#     ! queue ! videoconvert \
#     ! video/x-raw,format=BGR ! queue ! appsink")

if not cap.isOpened():
    raise IOError('Cannot open RICOH THETA')

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

Find JetPack version - Method #1

sudo apt-cache show nvidia-jetpack
[sudo] password for craig: 
Package: nvidia-jetpack
Version: 4.4.1-b50
Architecture: arm64
Maintainer: NVIDIA Corporation

Find JetPack Version - Method #2

cat /etc/nv_tegra_release 
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t210ref, EABI: aarch64, DATE: Fri Oct 16 19:44:43 UTC 2020

iketani_takuya · June 17, 2022, 9:29am

@craig
We apologize for our late reply, and we would like to report our current situation.
Although we downgrade our Jetpack from 4.5 to 4.3, the same error still appeared.
(gst_viewer worked correctly, but gst_loopback didn’t work)
We ran the command “gst-launch-1.0 thetauvcsrc num-buffers=150 ! queue ! h264parse ! fakesink”, it showed waring with “erroneous pipeline: no element thetauvcsrc”.
Since we haven’t read Basic tutorial thoroughly, we would like to debug the plugins later.

craig · June 17, 2022, 1:04pm

Did you install additional software after you installed JetPack? Do you have another version of libuvc installed?

Do you have the gstthetauvc.so installed in gstreamer-1.0 and not in gstreamer1.0?

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

What is the output of make?

cd thetauvc/
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ ls
gstglutils.c         gstthetatransform.h  gstthetauvcsrc.h  thetauvc.c
gstglutils.h         gstthetauvc.c        Makefile          thetauvc.h
gstthetatransform.c  gstthetauvcsrc.c     shader.h
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$ make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
craig@jetson:~/Documents/Development/gstthetauvc/thetauvc$

craig · July 6, 2022, 10:49pm

@iketani_takuya
I have streaming on my Jetson Nano working now with Jetpack 4.6.

Try uninstalling docker and make sure the camera is not mounted. Maybe something is interfering with the USB connection to the camera?


mkdir build
cd build/
cmake ..
sudo make install
...
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libuvc.so.0.0.6
-- Installing: /usr/local/lib/libuvc.so.0
-- Installing: /usr/local/lib/libuvc.so
-- Installing: /usr/local/include/libuvc/libuvc.h
-- Installing: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/libuvc.a
-- Up-to-date: /usr/local/include/libuvc/libuvc.h
-- Up-to-date: /usr/local/include/libuvc/libuvc_config.h
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcTargets-release.cmake
-- Installing: /usr/local/lib/cmake/libuvc/FindLibUSB.cmake
-- Installing: /usr/local/lib/cmake/libuvc/FindJpegPkg.cmake
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfigVersion.cmake
-- Installing: /usr/local/lib/pkgconfig/libuvc.pc
-- Installing: /usr/local/lib/cmake/libuvc/libuvcConfig.cmake

...
sudo ldconfig

sudo modprobe v4l2loopback
lsmod |grep v4l2
v4l2loopback           37127  0
 
ls -l /sys/devices/virtual/video4linux/
total 0
drwxr-xr-x 3 root root 0 Jul  6 15:18 video0

./gst_loopback 
start, hit any key to stop
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 


sudo apt install vlc

sudo apt-cache show nvidia-jetpack

[sudo] password for craig:

Package: nvidia-jetpack

Version: 4.6-b199

Architecture: arm64

Maintainer: NVIDIA Corporation

Installed-Size: 194

cat /etc/nv_tegra_release

# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021

works on /dev/video0 with vlc

Z1 is unmounted

gstthetauvc

did you do this?

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

and then restart the Jetson?

nicodax · December 14, 2022, 5:43pm

Hello all,

I am trying to develop a service for accessing cameras emitting from a Spot robot.

I plugged an RICOH THETA Z1 360 usb camera to the robot’s core and am trying to map the device to /dev/video0 (there are no other cameras on the core).

I followed this tutorial for setting up libuvc-theta and libuvc-theta-sample on the robot’s core.

I also followed this OpenCV Python demo and installed v4l2loopback on the robot.

When I try to run ./gst_loopback with the camera connected and in live stream mode, I get a Can't open THETA error.

Before testing this process on the robot’s core, I tried it on my local dev machine (ubuntu) and everything worked fine.

I am not very comfortable reading c code and i was not able to debug it myself…
It seems to be coming from the line 238 of the gst_viewer.c file:

res = uvc_open(dev, &devh);

Can someone explain what this does and how to output error logs from this command ?

Thanks in advance,

nicodax

PS: i edited the line 190 according to what i read in this forum’s discussion (qos=false and /dev/video0)

craig · December 14, 2022, 6:07pm

I do not have access to a spot. Is it a form of Linux?

consider gstthetauvc

Assuming you are using OpenCV, there are two ways to access the camera:

v4l2loopback will create a virtual video device /dev/video0 or /dev/video1 that you can use to access the device from OpenCV
use gstthetauvc to eliminate v4l2loopback

See this article on using gstthetauvc

Software - RICOH THETA Development on Linux

gstthetauvc is available here:

GitHub - nickel110/gstthetauvc: Gstreamer theta uvc plugin

The recommended approach is to use gstthetauvc instead of v4l2loopback.

The tutorial you are looking at is a bit old and most of the content was written before gstthetauvc was released. I should update it, but haven’t had the time. If there’s more interest in this type of deployment, I will prioritize the documentation update. If you are able, can you tell me what you are using the spot for and if it is a large deployment. This forum receives sponsorship from RICOH and I like to tell them that our documentation is in response to community demand. You can also send me a direct message if your deployment information is not appropriate to post in a public forum.

`Can't open THETA` error

This indicates that the device cannot be found on the spot. Does the spot already have another video camera at /dev/video0? If not, change line 190 to /dev/video0.

on the spot, can you see the THETA Z1 with gst_viewer ? Not gst_loopback.

$ ./gst_viewer -l
No : Product            : Serial    
 0 : RICOH THETA Z1     : 10010104

on the spot, can you see the v4l2loopback module loaded in the kernel?

$ lsmod
Module                  Size  Used by
uvcvideo               88565  0
bnep                   16562  2
zram                   26166  4
overlay                48691  0
spidev                 13282  0
v4l2loopback           37383  0

or

on the spot, can you see the dummy video device.

$ v4l2-ctl --info
Driver Info (not using libv4l2):
	Driver name   : v4l2 loopback
	Card type     : Dummy video device (0x0000)
	Bus info      : platform:v4l2loopback-000
	Driver version: 4.9.140
	Capabilities  : 0x85208003

On the spot, can you see the video device with lsusb

Feel free to post more questions.

We will get you up and running!

nicodax · December 15, 2022, 9:01am

Hello again!

Yes, I’m sorry I wasn’t thorough enough while describing my issue. Spot is a robot manufactured by Boston Dynamics and Spot’s CORE is an onboard computer running on Ubuntu 18.04 on the robot I am working with (not sure if this is a standard).

Thank you for the links about gstthetauvc, I will check it right after this message.

In a nutshell, the Spot robot I work with is owned by a research institute in the construction field. Right now, I am trying to build a web interface that connects to the robot and fetches the live video feeds from all its cameras (built-in or otherwise). Fetching the built-in ones was not a problem. Fetching this 360 RICOH USB cam is proving a bit more challenging. I only need it to be available as a /dev/video* device so that I can reuse my existing and working python scripts allowing the communication between the robot and another machine on the same network. I don’t think I have to go into more details about these scripts but if you are interested, I based my development off of this official demo.

About the Can’t open THETA error:

Indeed, ./gst_viewer -l does not see the RICOH THETA Z1 camera:

spot@SpotCORE:~/src/libuvc-theta-sample/gst$ ./gst_viewer -l
No : Product            : Serial    
 0 : (null)             : (null)

I can however see the v4l2loopback module loaded in the kernel (provided I load it after booting with the modprobe v4l2loopback command - I did not set it up to automatically load on boot yet);
I can also see the dummy video device:

spot@SpotCORE:~/src/libuvc-theta-sample/gst$ sudo v4l2-ctl --info
Driver Info (not using libv4l2):
	Driver name   : v4l2 loopback
	Card type     : Dummy video device (0x0000)
	Bus info      : platform:v4l2loopback-000
	Driver version: 4.15.18
	Capabilities  : 0x85208003

...

similarly, I can see the device with lsusb:

# The camera is in livestream mode:
spot@SpotCORE:~/src/libuvc-theta-sample/gst$ lsusb
Bus 002 Device 003: ID 05ca:2715 Ricoh Co., Ltd 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
# The camera went back to sleep:
spot@SpotCORE:~/src/libuvc-theta-sample/gst$ lsusb
Bus 002 Device 004: ID 05ca:036d Ricoh Co., Ltd 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Thanks a lot for the swift response sir, I was not expecting such availability.

Tell me if you are looking for a more specific description of what it is I am trying to do.

Good day to you,

nicodax

nicodax · December 15, 2022, 9:08am

Sorry for the second message, I just started reading the links you sent and I realised I was not clear enough (again) on my description.

The scripts I mentioned above are packaged into what Boston Dynamics calls an extension. Essentially it is a Docker container running a service.

For making the video devices available to the container, I have to run it with the --device argument.

As it stands, but I might have to change all of this, I need the device to be available as a video device otherwise I don’t know how to access the video feed.

I’ll keep reading the links you provided of course but I have a feeling this is not the solution I am looking for.

Thanks again,

nicodax

Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

nvdec, gldownload, glimagesink works with thetauvcsrc

gstthetauvc on Jetson Nano

make

install

use

result

OpenCV Test

Find JetPack version - Method #1

Find JetPack Version - Method #2

works on /dev/video0 with vlc

Z1 is unmounted

gstthetauvc

consider gstthetauvc

Can't open THETA error

`Can't open THETA` error