Bad frame capture with Theta Z1 and v4l2loopback

Hi all,

I have followed a lot of guides and finally made it so that I could get the gst_loopback working using the following. (I am using an x86 with Ubuntu 20.04.4 LTS)

"decodebin ! autovideoconvert ! "
"video/x-raw,format=BGRx ! identity drop-allocation=true !"
"v4l2sink device=/dev/video2 qos=false sync=false"

When I check the format provided on the v4l2loopback it seems ok, but I am not sure.

$ v4l2-ctl -d /dev/video2 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'BGR4' (32-bit BGRA/X 8-8-8-8)
		Size: Discrete 3840x1920
			Interval: Discrete 0.033s (30.000 fps)

Running this command, to output the video.

$ gst-launch-1.0 v4l2src device=/dev/video2 ! decodebin ! autovideoconvert ! xvimagesink

I get this result. The video is live but very distorted.

My goal is to do frame capture in OpenCV and tracking persons. In order to get VideoCapture to work with OpenCV in Python I have to use this in gst_loopback

"decodebin ! autovideoconvert ! "
"video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=I420 ! identity drop-allocation=true ! "
"v4l2sink device=/dev/video2 qos=false sync=false"

I still get a distorted image, but it looks different.

I have tried a lot of different things but now I am stuck. Any help is appreciated! :slight_smile:

Do you need to convert it to BGRx? What is the error when it is

video/x-raw,format=I420

Have you tried using this project?

I am using Ubuntu 20.04 on x86 as well. What video card do you have and are you using the iGPU or the video card?

Hi Craig,

Yes, the project you are referring to is the same I am using to feed the USB video stream into the v4l2loopback device. When I use the settings

video/x-raw,format=I420

I get this error.

$ ./gst_loopback 
start, hit any key to stop
Error: Internal data stream error.
stop

I am using an older Fujitsu laptop with an Nvidia Quadro K1100M/PCIe/SSE2 graphics chip. It has an I7 CPU and it is the 64-bit version of Ubuntu 20.04.4 LTS.

As a test, can you disable the discrete graphics card and run the gst_loopback program against the i7 internal GPU?

gst_loopback and v4l2linux may not work with the Quadro.

If possible, try this witht he example for OpenCV

Running Z1 on April 20, 2022

craig@craig-desktop:~/Development/ricoh/libuvc-theta-sample/gst$ sudo modprobe v4l2loopback
craig@craig-desktop:~/Development/ricoh/libuvc-theta-sample/gst$ lsmod |grep v4l2
v4l2loopback           40960  0
$ ./gst_loopback 
start, hit any key to stop

VLC test

successful

image

gst-launch-1.0 test

successful

pipeline

gst-launch-1.0 v4l2src device=/dev/video0 ! decodebin ! autovideoconvert ! xvimagesink

Device Firmware and model info

$ theta --info

THETA Device Info
==================
Model: RICOH THETA Z1
  manufacturer: Ricoh Company, Ltd.
  serial number: '10010104'
  device version: 2.01.0
  extension ID: 0x00000006
  image formats supported: 0x00000005
  extension version: 0x006e

opencv

import cv2

cap = cv2.VideoCapture(0)

# Check if the webcam is opened correctly
if not cap.isOpened():
    raise IOError("Cannot open webcam")

while True:
    ret, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.25, fy=0.25, interpolation=cv2.INTER_AREA)
    cv2.imshow('frame', frame)


    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

canny

Hi,

I installed the GStreamer Theta UVC plugin. I tried running below command.

gst-launch-1.0 thetauvcsrc mode=2K ! queue ! h264parse ! decodebin ! queue ! glimagesink sync=false

Displays the same distorted picture. :frowning:

Can you disable the discrete NVIDIA Quadro K1100M graphics card on your system using the BIOS on your laptop as a test and use the onboard GPU of the i7 CPU as a test? Or, does your i7 CPU not have an integrated GPU?

What NVIDIA driver are you using on Ubuntu 20.04? Are you running Wayland or X11?

The libuvc driver works for me on both Wayland and X11, but I think most people use it on X11.

checking driver version with nvidia-smi

checking X11 or Wayland with echo $XDG_SESSION_TYPE
image

verify that gst_viewer finds the camera

use lsusb to verify that device is 2715 (Z1)

gst_loopback test

Using Python3 with OpenCV 4.5.5

Hi,

Thank you for you support! So, today I tried doing this on a Windows machine. Using the Theta UVC driver I got it to work. So at least I know that the camera is fine.

The BIOS on the laptop is locked so I cannot disable it in the BIOS but I could switch to the Intel graphics chip using the Nvidia X Server Settings app in Ubuntu and PRIME Profiles settings.

Following your second last post, I got stuck on ./gst_loopback because it seemed to just hang and “start, hit any key to stop” was never displayed.

I could not try
$ theta --info
Where can I find that command?

Maybe I should try to install Ubuntu again? Is version 18 better than 20 for trying to get this to work? You clearly got it to work with 20…

theta --info is the same as ptpcam --info if you have that installed. I just wanted to show the firmware version. Can you post the firmware version you are using on the Z1?

You can use the desktop app on windows to upgrade the firmware of the Z1 if you haven’t done that already.

I do not think that you will benefit from reinstalling Ubuntu 18.04.

I think the majority of the people are running 20.04 unless you are running a ROS variant and need to run an older version of ROS due to existing programs from prior researchers.

I am likely going to upgrade to Ubuntu 22.04 Jammy Jellyfish now that it is released.

instead of reinstalling Ubuntu, my recommendation is to look into:

  1. Does v4l2loopback work on the Fujitsu laptop on Ubuntu 20.04 (what you have installed now) with a normal USB webcam running H.264 (like a normal logitech webcam, not a THETA Z1)
  2. verify that you are running the newest firmware in the Z1 (you can use the Windows desktop app Download | RICOH THETA)
  3. if you are still stuck after the firmware upgrade, consider running ubuntu on a separate disk drive using a different computer with a non-Quadro GPU. The Quadro is a high-end GPU, but as it is less common, we have less tests with it. Other people have posted about problems with v4l2loopback and Quadro graphics cards. I am suggesting a separate USB drive so that you don’t mess up a “work” Windows machine by re-partitioning the drive

Alternately, if someone in your lab has a Linux machine with a different hardware configuration, maybe you can borrow it so that you can try and isolate if the problem is with the Fujitstu hardware configuration with v4l2loopback?

Sorry to hear about all your problems. You’re not alone. If you search on this forum, you’ll see that certain hardware/software configurations have been problematic for other researchers, primarily with v4l2loopback.

Hi Craig,

Success at last! I gave it a go again on the Fujitsu with the new Ubuntu 22.04 LTS version (full install). I then installed libuvc-theta, libuvc-theta-sample, gstthetauvc, v4l2loopback and all necessary prerequisites. No changes other than that the v4l2loopback uses /dev/video2 on the Fujitsu.

I could not get VLC to work and the log didn’t reveal anything more:

Your input can't be opened:
VLC is unable to open the MRL 'v4l2:///dev/video2'. Check the log for details.

So, I moved on to test if the gst-launch-1.0 might work and it did! I also got the OpenCV Python test script to work, as well as your canny-edge.py script.

The Ubuntu installed Nvidia drivers are active and works.

Some screenshots:

Craig, thank you for all your extensive tips! :slight_smile:

2 Likes

Wow! Thanks for the update and congratulations on your determination to get this thing up and running. Nice.

Is the NVIDIA driver version 390.147 the newest version that runs with the Quadro?

I think the newest NVIDIA Driver out is 510.60.02.

Congratulations again. You should go out and celebrate and drink a beer. Man, what a journey. :slight_smile:

:beer: :beers: :clinking_glasses: :tumbler_glass:

Most people would have given up. You totally got my respect for keeping at the problem. And, you got it!

1 Like