Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

Hi Craig,
I have a new Theta z and I am trying to stream video from theta Z to the laptop using Python OpenCV and integrate on the ROS later. So far I am able to run stream the live video using gst_viewer. However, this doesn’t meet our requirement. Could you please let us know if it is possible to stream directly without using additional hardware.

I have followed this instruction ros_theta_z and ros_theta_s.
`$ v4l2-ctl --list-devices
Dummy video device (0x0000) (platform:v4l2loopback-000):
/dev/video1

BisonCam, NB Pro: BisonCam, NB (usb-0000:00:14.0-6):
/dev/video0`

As I am not able to see the theta z on my device lists, Could you please help me how to show theta z on the video device list. Thanks in advance!!

What version of Ubuntu/ROS are you using?

It’s possible to stream the Z1 to Linux x86 and NVIDIA Jetson without additional hardware.

$ v4l2-ctl --list-formats-ext --device  /dev/video1
ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'YU12' (Planar YUV 4:2:0)
        Size: Discrete 1920x960
            Interval: Discrete 0.033s (30.000 fps)

NOTE: your resolution will be in 4K. I have mine set to 2K for testing.

Did you go to the site below, enter an email and review the documentation on v4l2loopback?

RICOH THETA360.guide Independent developer community

If you have problems, please post again.

Tried this and changed video99 to video0. The file raw.hfyu is getting larger very slowly, about 10 seconds for 1.2Mb. And after I use Ctrl+C to end the gst-launch-1.0, I cannot open the raw.hfyu.

BTW, I’m using the Intel NUC mini PC(x86 and no Nvidia GPU) as it is the only device that I can use gst_loopback and gst_viewer. The gst_viewer runs smoothly and gst_loopback runs with less than 0.1fps(10s/1frame).

So frustrating for theta Z1, I might use theta S instead.

Did the file not open with VLC on Linux?

In the video below, I completely purged the NVIDIA drivers from my system that has an NVIDIA graphics card in it and covered the entire process using X11 nouveau. As a further test, I used Wayland and not X11. You don’t need to use Wayland, but the build process for libuvc-theta and libuvc-theta-sample may help. I couldn’t use gst_loopback with Wayland in my first test. I don’t recommend you use Wayland at the moment.

Dear Craig

I gave up Theta Z1 with my NUC. Now by using my old Theta S, I can successfully output image and video using ROS + OpenCV. But there is another problem.

The video is dual-fisheye. It cannot be converted to equirectangular by RicohTheta Basic. It stuck in 100% no matter how small the video is.

The image or video have no metadata. The image can be converted but video cannot open.
Do you have any idea how I can transform these dual-fisheye video into equirectangular?

Oh. I found that changing encoding method of OpenCV from mp4v to avc1 solved the problem and theta software can open the video.

But the problem is still the lack of metadata, which leads to stitching error in the middle

(theta is placed side-down rather than straight for less vibration.

Do you have any information about how I can get the metadata from theta S while livestreaming

1 Like

nickel110 updated gstthetauvc

I have not tested it yet.

Have you looked at this ROS package? I’m not sure if it has the metadata, but it may have some tools that can help you.

I am having exactly same issue on Jetson nano B01 version. Any luck with this ?

Jetson Nano B01
Ubuntu 18.04.5 LTS

My end goal is to live stream video in browser and eventually run tf model .

I think the original poster got it working, but was having problems saving the frames to disk.

Are you getting an error message?

What model THETA are you using V or Z1?

I am using Jetson Nano B01 with no problems using JetPack 4.4, which I think is Ubuntu 18.04.

Hi Craig,

Thanks for replying back, I tried on two systems two different issues

Ricoh Theta Z1 (51GB)

Laptop - Ubuntu 18.04 with build-in camera

  1. Followed instructions to install all the required lib

  2. when I run ./gst_viewer it works

  3. when i run ./gst_loopback

    ./gst_loopback
    start, hit any key to stop
    Error: Device ‘/dev/video1’ is not an output device.
    stop

  4. ls -ls /dev/vid*

     0 crw-rw----+ 1 root video 81,   0 Jun  7 11:42 /dev/video0
     0 crw-rw----+ 1 root video 81,   1 Jun  7 11:42 /dev/video1
    
     Note : This does not identify Ricoh 
    
     Step 1: verify usb device ID
     Bus 002 Device 020: ID 05ca:2715 Ricoh Co., Ltd 
    
     Step 2: verify kernel module
    
     Module                  Size  Used by
     v4l2loopback           45056  0
    
    
     Step 3: verify device
    
     ls /dev/vide*
     /dev/video0  /dev/video1  /dev/video2
    
     added  video2
      v4l2-ctl --list-formats-ext --device  /dev/video2
     ioctl: VIDIOC_ENUM_FMT
    

now when I run VLC it only shows the first frame and does not show the video


import cv2

cap = cv2.VideoCapture(2)

Check if the webcam is opened correctly

if not cap.isOpened():
raise IOError(“Cannot open webcam”)

while True:
ret, frame = cap.read()
frame = cv2.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
cv2.imshow(‘Input’, frame)

c = cv2.waitKey(1)
if c == 27:
    break

cap.release()
cv2.destroyAllWindows()


When I use open cv simple code ( just to capture at index2 ) I get below error

python3 simple.py
[ WARN:0] global /tmp/pip-req-build-1syr35c1/opencv/modules/videoio/src/cap_v4l.cpp (1004) tryIoctl VIDEOIO(V4L2:/dev/video2): select() timeout.
Traceback (most recent call last):
File “simple.py”, line 11, in
frame = cv2.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
cv2.error: OpenCV(4.5.1) /tmp/pip-req-build-1syr35c1/opencv/modules/imgproc/src/resize.cpp:4051: error: (-215:Assertion failed) !ssize.empty() in function ‘resize’

Second System
System 2 jet set nano
Ubuntu 18.04 LTS
Getting below error

./gst_loopback
start, hit any key to stop
Error: Cannot identify device ‘/dev/vidoe0’.
stop

Step 1: verify usb device ID
Bus 002 Device 004: ID 05ca:2715 Ricoh Co., Ltd

Step 2: verify kernel module
Module Size Used by
v4l2loopback 42767 0

Step 3: verify device
king@king-desktop:~/dev/ricoh/libuvc-theta-sample/gst$ v4l2-ctl --list-formats-ext --device /dev/video0
ioctl: VIDIOC_ENUM_FMT

./gst_loopback
start, hit any key to stop
Error: Cannot identify device ‘/dev/vidoe0’.
stop

Please let me know if you need any more information. I needed to achieve

  1. Live feed this into my angular application using flask or anything else
  2. Take this live feed and run models.

Bharat

Did you load the kernel module for v4l2loopback and can you see it?

https://codetricity.github.io/theta-linux/software/

$ lsmod
Module                  Size  Used by
bnep                   16562  2
zram                   26166  4
overlay                48691  0
spidev                 13282  0
v4l2loopback           37383  0
nvgpu                1579891  18
bluedroid_pm           13912  0
ip_tables              19441  0
x_tables               28951  1 ip_tables
craig@jetson:/etc/modules-load.d$ 

You can test it with VLC

$ sudo modprobe v4l2loopback
$ cd path_to_gst_loopback_directory
$ ./gst_loopback
$ cvlc v4l2:///dev/video0
VLC media player 3.0.9.2 Vetinari (revision 3.0.9.2-0-gd4c1aefe4d)
[0000556fc2bd6db0] dummy interface: using the dummy interface module...

Compile and install

$ git clone https://github.com/umlaeute/v4l2loopback.git
$ cd v4l2loopback
$ make 
$ sudo make install
$ sudo depmod -a

These are older videos

Jetson Nano build screencast

Jetson Nano vl2loopback screencast

OpenCV and VLC Test

You should cut down the resolution to 2K for Jetson Nano. It has too high latency at 4K on the Nano.

Thanks for getting back .

On Ubuntu laptop

Yes, I can see the kernel module

$ lsmod
Module                  Size  Used by
uas                    28672  0
cdc_acm                40960  0
usb_storage            77824  1 uas
rndis_wlan             61440  0
rndis_host             20480  1 rndis_wlan
cdc_ether              20480  1 rndis_host
v4l2loopback           45056  1
ccm                    20480  6

When testing with VLC

$ sudo modprobe v4l2loopback
   gst$ ./gst_loopback 
   start, hit any key to stop
$ cvlc v4l2:///dev/video2
VLC media player 3.0.8 Vetinari (revision 3.0.8-0-gf350b6b5a7)
[000055c9350c7ad0] dummy interface: using the dummy interface module...
#VLC OPENS BUT HANGS ON FIRST FRAME

If I do CTR+C it stops.

If I close the VLC window I get

[00007f61a403c300] xcb_window window error: X server failure

OpenCV still has the same issue

On Jetson Nano

Yes, I see it

gst$ lsmod
Module                  Size  Used by
v4l2loopback           42767  0
$ sudo modprobe v4l2loopback
$ ./gst_loopback 
    start, hit any key to stop
    Error: Cannot identify device '/dev/vidoe0'.
   stop
$ ls /dev/video0 
/dev/video0

NOTE: gst_viewer does not have any issues

There are two separate problems.

On the jetson nano, there is a typo in the device name. You have /dev/vidoe0. It would be /dev/video0. The typo is likely in the C code.

on the laptop, you are likely using software graphics acceleration, not hardware. If it’s a laptop, you can first try and disable the discrete GPU and use the onboard GPU on your Intel CPU.

For the x86 problem, you need to install gstreamer1.0-plugins-bad to get support for the discrete NVIDIA card.

You can try installing all the gstreamer plug-ins with:

$ sudo apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio libgstreamer-plugins-base1.0-dev

You can try using nvdec with gstreamer on x86.

Optimization - RICOH THETA Development on Linux

You do not need to do this on the Jetson as NVIDIA JetPack ships with support for NVIDIA hardware on the Jetson.


If you are still stuck on x86 with the loopback, try using this plug-in to gstreamer.


BTW, after you get things running, please share more about your project. I can pass the information on to the developer of the driver. He’s interested in what people are doing with Linux streaming.

Hi Craig,

Please see below updates (and thanks again for help and quick response)

Jetson nano.
I got it working after I fixed the typo (I guess I need to stop working late nights :slight_smile: )
VLC was crashing so had to put the below fix

I got both VLC and opencv working but they are wayyyyy to slowwwwwwwww . Any pointers to make it fast

“You should cut down the resolution to 2K for Jetson Nano. It has too high latency at 4K on the Nano” will be try this next

Laptop
My edge computer does not have GPU it only has CPU .

I will try this in the evening and let you know how it went

The Nano does not have enough hardware resources to process video at 4K. You may have better success with Xavier. For test on the Nano, set the resolution to 2K per the instructions below.

Help - RICOH THETA Development on Linux

In gst_viewer.c, change line 248 from THETAUVC_MODE_UHD_2997 to THETAUVC_MODE_FHD_2997.

Refer to thetauvc.c#L55 for definition.

Thanks for the fix to VLC. I was not aware of this. I’ll try it out. :slight_smile:


If the laptop does not have a discrete GPU, you can likely get it running with the gstreamer-bad plug-ins for gstreamer. The default gstreamer on x86 may not have NVIDIA hardware acceleration. The frame is likely stopping because it is trying to use software processing.

What CPU do you have in the laptop?

If you are ambitious, you can also try the gstreamer plug-in by nickel110. Please post your results if you decide to use the thetauvc plug-in for gstreamer.

This is my laptop,

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 8
CPU family: 6
Model name: Intel(R) Core™ i7-8665U CPU @ 1.90GHz
CPU MHz: 3499.832
CPU max MHz: 4800.0000
CPU min MHz: 400.0000
NUMA node0 CPU(s): 0-7

I have another edge device, where I can only install once I am comfortable on Nano and my laptop.

Motherboard = i5 Intel® 8th Gen (Whiskey lake-U) Core™ CCG Lifecycle
Memory (RAM) = 16 GB DDR4 2666 MHz
Primary storage = 512 GB SSD
Operating system = Ubuntu Desktop 18.04 LTS 64-bit

How do I install the plugin, I did not see any documentation in git( Sorry new to this )

The laptop you are using should work.

I believe the gstreamer plug-in may be a better route. However, no one has documented how to use it. I have not used it myself. The developer, nickel110, is the same developer that wrote the libuvc-theta and libuvc-theta-sample projects.

Based on the high-quality of his other open source projects, I think it is likely that the gstreamer plug-in does work.

the information below is based on an email from the open source developer. If you are willing to try this approach and document it here, I can email the developer and point out your post if you get stuck. This is a personal project of the developer. He’s often busy with his normal job and he’s also not a native English speaker. so, he doesn’t often post directly in this forum. But, he may be able to provide advice if we have a specific problem with the thetauvc plug-in he wrote for gstreamer.

If you give it a try, it may also motivate other people to try it. It’s likely that the plug-in solves other problems such as easier use of multiple THETA cameras on the same computer.

If OpenCV is built with gstreamer backend enabled, VideoCapture::open() accepts the gstreamer pipeline description as its argument like

  VideoCapture cap;
  cap.open("thetauvcsrc ! h264parse ! decodebin ! videoconvert ! appsink");

opencv/cap_gstreamer.cpp at 1f726e81f91746e16f4a6110681658f8709e7dd2 · opencv/opencv · GitHub

you can capture image directly from gstreamer.

Please note that the standard image format on OpenCV is “BGR” which is not supported by most hardware assisted colorspace converter plugins, thus, you have to use software converter.

Since recent OpenCV accepts I420 or NV12 as input format for VideoCapture, you can capture without color conversion and convert using OpenCV if necessary.

It’s possible that gstreamer could be used with the ROS OpenCV camera driver if OpenCV is built with the gstreamer backend enabled.

  • Makefile in the gstthetauvc does not have “install” target. Please copy the plugin file (gstthetauvc.so) into appropriate directory. If you copied it into directory other than gstreamer plugin directory, you have to set the directory to the GST_PLUGIN_PATH environment variable.
  • The plugin has several properties to specify resolution and THETA to use(if multipe THETA are connected to the system).
    Run “gst-inspect-1.0 thetauvcsrc” for detail.
  • For Ubuntu 20 and Jetson users, OpenCV packages from Ubuntu official or nvidia have gstreamer backend.
  • For Ubuntu 18 users, official OpenCV packages do not have gstreamer backend and the version(3.2) is too old to use with the gstreamer. If you want to use on Ubuntu 18, build the latest OpenCV from the source, or AT YOUR OWN RISK, you can use unofficial OpenCV binary packages for Ubuntu 18 like

opencv-4.2 : “cran” team

Sure, I will definitely try it and document it. I got stuck on this step

  1. Cloned repo
gstthetauvc/thetauvc$ ls -l
total 100
-rw-r--r-- 1 y5123 y5123  4562 Jun  8 21:26 gstglutils.c
-rw-r--r-- 1 y5123 y5123  1479 Jun  8 21:26 gstglutils.h
-rw-r--r-- 1 y5123 y5123 18331 Jun  8 21:26 gstthetatransform.c
-rw-r--r-- 1 y5123 y5123  2233 Jun  8 21:26 gstthetatransform.h
-rw-r--r-- 1 y5123 y5123  1514 Jun  8 21:26 gstthetauvc.c
-rw-r--r-- 1 y5123 y5123 22585 Jun  8 21:26 gstthetauvcsrc.c
-rw-r--r-- 1 y5123 y5123  2262 Jun  8 21:26 gstthetauvcsrc.h
-rw-r--r-- 1 y5123 y5123   799 Jun  8 21:26 Makefile
-rw-r--r-- 1 y5123 y5123 14119 Jun  8 21:26 shader.h
-rw-r--r-- 1 y5123 y5123  7774 Jun  8 21:26 thetauvc.c
-rw-r--r-- 1 y5123 y5123  2596 Jun  8 21:26 thetauvc.h
  1. make
/gstthetauvc/thetauvc$ make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
  1. after make
gstthetauvc/thetauvc$ ls -ls
total 224
  8 -rw-r--r-- 1 y5123 y5123   4562 Jun  8 21:26 gstglutils.c
  4 -rw-r--r-- 1 y5123 y5123   1479 Jun  8 21:26 gstglutils.h
 20 -rw-r--r-- 1 y5123 y5123  18331 Jun  8 21:26 gstthetatransform.c
  4 -rw-r--r-- 1 y5123 y5123   2233 Jun  8 21:26 gstthetatransform.h
  4 -rw-r--r-- 1 y5123 y5123   1514 Jun  8 21:26 gstthetauvc.c
**120 -rwxr-xr-x 1 y5123 y5123 119800 Jun  8 21:43 gstthetauvc.so**
 24 -rw-r--r-- 1 y5123 y5123  22585 Jun  8 21:26 gstthetauvcsrc.c
  4 -rw-r--r-- 1 y5123 y5123   2262 Jun  8 21:26 gstthetauvcsrc.h
  4 -rw-r--r-- 1 y5123 y5123    799 Jun  8 21:26 Makefile
  4 drwxr-xr-x 2 y5123 y5123   4096 Jun  8 21:43 obj
 16 -rw-r--r-- 1 y5123 y5123  14119 Jun  8 21:26 shader.h
  8 -rw-r--r-- 1 y5123 y5123   7774 Jun  8 21:26 thetauvc.c
  4 -rw-r--r-- 1 y5123 y5123   2596 Jun  8 21:26 thetauvc.h
  1. Ran the command and did not get what was expected
gstthetauvc/thetauvc$ gst-inspect-1.0 thetauvcsrc
No such element or plugin 'thetauvcsrc'

any advice?

As they may take a while to find a fix for, can you try the x86 again and verify that you set qos=false

/dev/video0 freezes on x86

Change line 190 of gst_viewer.c.

if (strcmp(cmd_name, "gst_loopback") == 0)
    pipe_proc = "decodebin ! autovideoconvert ! "
        "video/x-raw,format=I420 ! identity drop-allocation=true !"
        "v4l2sink device=/dev/video0 qos=false sync=false";