Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

[nicodax]

Hello again,

I have an update on the situation.

The Spot CORE has an output HDMI port and I used it to connect an external monitor for testing purposes and shazam! it does work this way. Is there any reason for this ?

I checked again and it does not work when the monitor is plugged out.

I read somewhere that Xorg would not start unless a monitor was plugged in, so I tried starting it manually to see if that would solve the issue, but it doesn’t.

Maybe someone with better knowledge would know what it means ?

I juste checked the thetauvc.c file and indeed both lines you pointed out were missing. I added them and ran make clean && make to rebuild it but it doesn’t seem to have done anything.

nicodax

[craig]

sorry about my earlier post. I got confused for a moment and thought you were using the THETA X instead of the THETA Z1. I deleted my earlier post.

I don’t have a good solution right now.

I don’t know about X.org monitor issues. Hopefully, someone else will be able to comment on this. You could also try an X.org-specific group on Reddit or stackoverflow. It seems like the core problem is using v4l2loopback with X.org without a monitor.

If it works with the monitor, maybe try a dummy HDMI plug as part of the debugging process and see if it works without a monitor and just the dummy plug?

Are you unable to use gstthetauvc ?

codetricity/theta-linux/blob/b7b0b5d1e928ea50c6cb005ed2b5a4d9abf66de6/code/canny_edge.py#L39

Does docker prevent you from using libusb, libuvc directly?

nickel110/gstthetauvc/blob/master/thetauvc/thetauvc.c

---

I do not use docker. I just searched for ideas about how to access USB devices on Docker and this post came up.

Docker container cannot access dynamically plugged USB devices

---

I’m reaching the limits of the knowledge that I can offer. However, feel free to keep posting as there are other people more knowledgeable than I am. It may take some time to get an answer. Just letting you know that my ideas may waste your time, so choose the best course.

[nicodax]

Hello all!

I fixed the issue. It was actually quite simple, I just did not know this was a thing.

Apparently, there is a llinux user group called plugdev that allows access to USB devices from an SSH session. I only had to run this command:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Now everything works fine.

Thanks a lot @craig for your availability, this meant a lot to me.

Take care!

[craig]

@nicodax

Apparently, there is a llinux user group called plugdev that allows access to USB devices from an SSH session. I only had to run this command:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Thank you for posting the solution.

I was not aware of the plugdev group. This will help other Spot users.

have a great holiday seasoon.

[craig]

tip for gstthetauvc on Jetson Xavier NX

credit alex-uam
reference on github issue

problem

I have compiled the gstthetauvc plugin and am attempting to run it on a Jetson Xavier NX, however I receive the error :

ERROR: from element /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0: Found 1 Theta(s), but none available.

I have successfully gotten it running on a Jetson Xavier AGX, as well as an Ubuntu desktop PC. Just wondering if there are any obvious steps that I’ve missed for the NX? Possibly permission issues?

solution

We managed to find the solution. Turns out it was as simple as setting an environmental variable in bash:

export LD_LIBRARY_PATH=/usr/local/lib

For whatever reason, this is automatically set on Intel based Ubuntu, and the Xavier AGX series board, but not the NX!

---

other

on my (craig) system, this is my /etc/ld.so.conf.d/libc.conf

 cd /etc/ld.so.conf.d/
craig@jetpack-4:/etc/ld.so.conf.d$ ls
aarch64-linux-gnu.conf             fakeroot-aarch64-linux-gnu.conf
aarch64-linux-gnu_EGL.conf         libc.conf
aarch64-linux-gnu_GL.conf          nvidia-tegra.conf
cuda-10-2.conf                     vpi1.conf
fakechroot-aarch64-linux-gnu.conf
craig@jetpack-4:/etc/ld.so.conf.d$ cat libc.conf 
# libc default configuration
/usr/local/lib

[buburider]

Hi all,

I’m working with Theta X and I use gstthetauvc plugin for capturing the video stream. I had some issues using it. Fortunately, I could make it work by using 30 fps in the format settings (instead of 29) (thetauvc_get_stream_ctrl_format_size function in thetauvc.c file).

Now, I created a docker image to ease the deployment of the streaming in several machines. However, I couldn’t make it work. Camera is detected but UVC returns an error of “No such device” when uvc_open() is executed. This is the command that I’m using to create the container:

docker run --network host --privileged --device '/dev/bus/usb' gst-xrlab

And this is the gstreamer pipeline that runs inside the container:

gst-launch-1.0 -v thetauvcsrc ! queue ! h264parse ! decodebin ! queue ! autovideosink sync=false

Does anyone stream using a docker image? Does he/she find this kind of issue? I’d really appreciate any help. Thanks in advance.

KR,

 Juan

[craig]

did you see the post by @nicodax about the user permissions:

sudo usermod -aG plugdev [SPOT_CORE_USER]

Actually, now that I reread his post, I think he’s using libuvc-theta-sample

Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

Sorry for the second message, I just started reading the links you sent and I realised I was not clear enough (again) on my description. The scripts I mentioned above are packaged into what Boston Dynamics calls an extension. Essentially it is a Docker container running a service. For making the video devices available to the container, I have to run it with the --device argument. As it stands, but I might have to change all of this, I need the device to be available as a video device otherwi…

[buburider]

Thank you very much for the reply.

I’m afraid that the user that run the container is within plugdev group. Anyway, it’s weird that in my case camera is detected inside the container but it cannot be open. AFAIK, @nicodax had issues detecting the camera.

[craig]

I unfortunately have not tried the Linux streaming with docker. It seems like you have it working outside of docker, right?

Please post your system config such as Ubuntu 22.04 on x86 or ROS Noetic Ninjemys on Nvidia Xavier NX.

I do not have experience with Docker in general, so the questions below are likely irrelevant as you have streaming working outside of docker.

are /usr/local/lib/ and the location of gstthetauvc.so in your library load path?

You may have done something like this:

sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/

Note that there is a hyphen between gstreamer and 1.0.

[buburider]

Thanks a lot for the suppport.

The host of the docker image is Ubuntu 20.04 on x86_64 architecture.

In addition, I use the variable GST_PLUGIN_PATH to set where the plugin is located. Plugin is detected and working. This is the log that is generated with gstreamer using: GST_DEBUG=2,thetauvcsrc:6

0:00:00.688153989     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:208:gst_thetauvcsrc_set_property:<thetauvcsrc0> set_property
0:00:00.688169165     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:208:gst_thetauvcsrc_set_property:<thetauvcsrc0> set_property
Setting pipeline to PAUSED ...
0:00:00.689459545     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689466171     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.689476892     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689479785     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.689487561     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689489383     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.689498710     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:672:gst_thetauvcsrc_query:<thetauvcsrc0> query
0:00:00.689500063     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:684:gst_thetauvcsrc_query:<thetauvcsrc0> query CAPS

0:00:00.690952410     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:448:gst_thetauvcsrc_start:<thetauvcsrc0> start
0:00:00.690960576     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:449:gst_thetauvcsrc_start:<thetauvcsrc0> dev=-1 mode=0
0:00:00.760036006     8 0x55dc6da40c00 WARN             thetauvcsrc gstthetauvcsrc.c:493:gst_thetauvcsrc_start:<thetauvcsrc0> error: Found 1 Theta(s), but none available (No such device).
0:00:00.760414751     8 0x55dc6da40c00 WARN                 basesrc gstbasesrc.c:3468:gst_base_src_start:<thetauvcsrc0> error: Failed to start
0:00:00.760480896     8 0x55dc6da40c00 WARN                 basesrc gstbasesrc.c:3824:gst_base_src_activate_push:<thetauvcsrc0> Failed to start in push mode
0:00:00.760501442     8 0x55dc6da40c00 WARN                GST_PADS gstpad.c:1142:gst_pad_set_active:<thetauvcsrc0:src> Failed to activate pad
ERROR: Pipeline doesn't want to pause.
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0: Found 1 Theta(s), but none available (No such device).
Additional debug info:
gstthetauvcsrc.c(493): gst_thetauvcsrc_start (): /GstPipeline:pipeline0/GstThetauvcsrc:thetauvcsrc0
Freeing pipeline ...
0:00:00.761629738     8 0x55dc6da40c00 DEBUG            thetauvcsrc gstthetauvcsrc.c:270:gst_thetauvcsrc_finalize:<thetauvcsrc0> finalize

KR,

Juan

[buburider]

Hi again,

I tested a Jetson Nano having exactly the same issue that I got using docker. Some remarks that might be important:

• I got the same error using gstthetauvc plugin and libuvc-theta-sample: libuvc raised an error of “Not supported (code -12)” when uvc_open() ran.
• Same code and procedures worked fine in a laptop running Ubuntu 20.04.
• My camera is Theta X. From this thread, it seems that the same tools works without issues with Z1 and V.
• Same issue running as root. Just in case, I also added the login user to plugdev group (sudo usermod -aG plugdev USER).
• Camera was detected by uvc (of course, using lsusb and usb-devices as well).

I’d really appreciate any help. Thanks in advance!

Juan

[craig]

sorry that I can’t be more helpful, but I’m not able to figure out the source of the problem. Maybe other people can suggest a solution.

In the version of gstthetauvc you are using, is the USBPID_THETAX_UVC defined as 0x2717 in thetauvc.h?

nickel110/gstthetauvc/blob/2914a3ef2feb39a182a8cc505cfae0e2d8922f0a/thetauvc/thetauvc.h#L43

I can try and test the X with my Jetson Nano.

[craig]

Test on Jetson Nano, March 6, 2023.

both gstthetauvc and libuvc-theta-sample work with RICOH THETA X

THETA Device Info
==================
Model: RICOH THETA X
  manufacturer: Ricoh Company, Ltd.
  serial number: '14010001'
  device version: 1.41.0
  extension ID: 0x00000006
  image formats supported: 0x00000004
  extension version: 0x006e

Make sure the camera is not mounted by the OS.

Jetpack 4.6

cat /etc/nv_tegra_release 
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t210ref, EABI: aarch64, DATE: Mon Jul 26 19:20:30 UTC 2021

or using apt-cache

apt-cache show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.6-b199
Architecture: arm64
Maintainer: NVIDIA Corporation

opencv test

cat > sample.py
import cv2
# pipeline below worked on Jetson and x86
cap = cv2.VideoCapture("thetauvcsrc \
    ! decodebin \
    ! autovideoconvert \
    ! video/x-raw,format=BGRx \
    ! queue ! videoconvert \
    ! video/x-raw,format=BGR ! queue ! appsink")

# pipeline suggestion thanks to nickel110
# attempt to force hardware acceleration
# tested with NVIDIA 510.73 with old GTX 950 on Ubuntu 22.04
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! queue \
#     ! h264parse \
#     ! nvdec \
#     ! gldownload \
#     ! queue \
#     ! videoconvert n-threads=0 \
#     ! video/x-raw,format=BGR \
#     ! queue \
#     ! appsink")

# NVIDIA Jetson
# cap = cv2.VideoCapture("thetauvcsrc \
#     ! nvv4l2decoder \
#     ! nvvidconv \
#     ! video/x-raw,format=BGRx \
#     ! queue ! videoconvert \
#     ! video/x-raw,format=BGR ! queue ! appsink")

if not cap.isOpened():
    raise IOError('Cannot open RICOH THETA')

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

run it

python sample.py 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=1, duration=-1

full build of gstthetauvc, installation and test

using RICOH THETA X

make
mkdir ./obj
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvc.o gstthetauvc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/gstthetauvcsrc.o gstthetauvcsrc.c
cc -fPIC -g -Og -pthread -I/usr/local/include -I/usr/include/gstreamer-1.0 -I/usr/include/glib-2.0 -I/usr/lib/aarch64-linux-gnu/glib-2.0/include -I/usr/include/libusb-1.0 -c -o obj/thetauvc.o thetauvc.c
cc -shared -o gstthetauvc.so obj/gstthetauvc.o obj/gstthetauvcsrc.o obj/thetauvc.o  -L/usr/local/lib -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -luvc -lusb-1.0
craig@jetpack-4:~/Development/gstthetauvc/thetauvc$ ls
gstglutils.c         gstthetauvc.c     Makefile    thetauvc.h
gstglutils.h         gstthetauvc.so    obj
gstthetatransform.c  gstthetauvcsrc.c  shader.h
gstthetatransform.h  gstthetauvcsrc.h  thetauvc.c
craig@jetpack-4:~/Development/gstthetauvc/thetauvc$ sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0/
craig@jetpack-4:~/Development/gstthetauvc/thetauvc$ gst-launch-1.0 thetauvcsrc mode=4K ! queue ! h264parse ! nvv4l2decoder ! queue ! nv3dsink sync=false
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
XIO:  fatal IO error 0 (Success) on X server ":0"
      after 844 requests (844 known processed) with 0 events remaining.

action shots

• THETA X firmware 1.41
• gstthetauvc
• NVIDIA Jetson Nano running Jetpack 4.6

---

gstthetauvc update on March 4, 2023

framerate negotiation updated.

Revise framerate negotiation.

[buburider]

Thank you for your test. Great news! So, I know that I must be able to make it work now. Although I haven’t done yet. Theta X pid is defined in my source code. If it wasn’t, camera wouldn’t be detected.

But, I don’t understand what you said:

@craig

Make sure the camera is not mounted by the OS.

Camera is already mounted by the OS. At least, I only plug the camera and it appears as USB device. I don’t know how to do it in another way.

user@jetson:~$ lsusb
Bus 001 Device 003: ID 05ca:0373 Ricoh Co., Ltd 
user@jetson:~$ ls -l /dev/bus/usb/001/003 
crw-rw-r--+ 1 root plugdev 189, 2 mar  6 17:19 /dev/bus/usb/001/003

KR,

  Juan

[craig]

@buburider

Camera is already mounted by the OS. At least, I only plug the camera and it appears as USB device. I don’t know how to do it in another way.

If the camera is mounted as a filesystem, the stream may not work on the NVIDIA Jetson devices. See the following article.

Disable THETA Automount on Linux and NVIDIA Jetson for USB API

When using the RICOH THETA with the USB API, the camera may be inaccessible. With the RICOH THETA Z1 and ptpcam, you may see this type of error message. craig@jetpack-4:~$ theta --info THETA Device Info ================== ERROR: Could not open session! craig@jetpack-4:~$ craig@jetpack-4:~$ theta --capture Initiating capture... ERROR: Could not open session! craig@jetpack-4:~$ After manual unmount You can use manual unmount. [image] craig@jetpack-4:~$ theta --info THETA Device Info =…

this older article may also be relevant.

RICOH THETA API Over USB Cable - (X, Z1, SC2, V, S, SC models)

Using gphoto2 on a Raspberry Pi to control the RICOH THETA Over a USB Cable I have gphoto2 working on the Raspberry Pi. This is installable with apt and is easier to use than ptpcam. [image] In the example below, I set the shutter chirp to 30% of sound volume and then verify that the setting in correct. pi@raspberrypi ~/Development/libptp2-1.2.0 $ gphoto2 --set-config=502c=30 pi@raspberrypi ~/Development/libptp2-1.2.0 $ gphoto2 --get-config=502c Label: PTP Property 0x502c Type: MENU Curren…

you can try kill the gphoto2-volume-monitor and spawner

[biviel]

@buburider

@craig

Make sure the camera is not mounted by the OS.

Camera is already mounted by the OS. At least, I only plug the camera and it appears as USB device. I don’t know how to do it in another way.

hi, @craig ,
it may not be relevant to this, but while I was doing in past experiments with USB connected Theta Z1, I noticed some difference when on Z1, the “Live” mode was turned on or off. I do not remember exactly, but I was having issues at that time, it may or may not be there at Theta X. But also how device was detected by my Raspberry Pi 4 running ubuntu, it was different depending on actual stat of camera (MODE button on Z1).

[craig]

The USB ID is going to be different if the camera is in still image mode. However, I think that @buburider has the camera in live streaming mode.

#define USBPID_THETAX_UVC 0x2717

Example When X Camera in Still Image mode

Device ID is 2718.

craig@craig-desktop:~$ lsusb
...
Bus 005 Device 002: ID 05ca:2718 Ricoh Co., Ltd RICOH THETA X
...

Example when in live streaming mode

device id is 2717

craig@craig-desktop:~$ lsusb
Bus 005 Device 003: ID 05ca:2717 Ricoh Co., Ltd RICOH THETA X

default behavior is to mount THETA X as a storage device

You must unmount it

[buburider]

As @craig said I’m using live streaming mode. So the USB ID is 0x2717.

I’ve just found the issue: my jetson computer had a previous libuvc library installed. So, instead of using libuvc-theta, the application loaded the native version. This is the reason why the same code was working in my laptop. In the laptop, the native version of libuvc didn’t exist. After resolving that, it works fine.

Thank you very much for the help. It’s really awesome to have this forum to share results and issues.

KR,

  Juan

[craig]

First, congratulations on getting it working!

Thank you for sharing the solution.

I’m sorry that I didn’t recommend this earlier. This is a known problem and I was just researching the issue last week when you asked the question. However, I didn’t make the connection between the previous libuvc and your environment. I’m sharing the additional information below to help other people with this problem. Thanks again for contributing your solution.

Found 1 Theta(s), but none available.

There is a note from the author of gstthetauvc

this was the original problem.

[craig]

@buburider

Thank you very much for the help. It’s really awesome to have this forum to share results and issues.

great to hear. If you ever talk to anyone at RICOH, please mention that this forum was useful. We’re dependent on continued sponsorship from RICOH to keep the independent forum going.

Also, I’ve just updated this document with the error message first reported and your solution.

Help - RICOH THETA Development on Linux

very glad that you got it working. I respect that you kept going even when the source of the problem was tough to trace.