Ricoh Theta V: Livestreaming with Jetson Xavier, ROS, OpenCV, NUC

I can’t get the gstream pipeline to work directly from the terminal… this might be the issue

gst-launch-1.0 v4l2src device=/dev/video7 ! video/x-raw,framerate=30/1 ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Did ./gst_loopback work fine on a NUC?

@ bot-lin Yes, I am using Intel NUC10i7fnh, and it worked great with the following code in the gstviewer file:

if (strcmp(cmd_name, "gst_loopback") == 0)
		pipe_proc = "decodebin ! autovideoconvert ! "
			"video/x-raw,format=I420 ! identity drop-allocation=true !"
			"v4l2sink device=/dev/video7  qos=false sync=false";
	else
		
		pipe_proc = " decodebin ! autovideosink sync=false qos=false";

I have tried Jetson Nano, and I met the same problem as yours. The JetPack version which I used is 4.6 but it was 4.4 posted herer. I am trying to recover the system to 4.4 and see what is going on.

I did a series of tests with a Jetson Nano yesterday streaming with a THETA V running firmware 3.70.1.

I can’t replicate the problem.

  • OpenCV 4.4.0 works with Python
  • DetectNet works
  • gst-launch-1.0 works

My configuration

  • L4T 32.4.4
  • power is from barrel connection with external fan
  • using 2K resolution for DetectNet and OpenCV as the 4K stream slows down the Nano
  • RICOH THETA V with firmware 3.70.1 connected with microUSB cable

I’ll try this with Jetpack 4.6 later

Thanks, Craig, how can I check the firmware version of my Theta V?

connect the camera with a USB cable to either a Mac or Windows computer.

https://support.theta360.com/en/download/

I confirm that my firmware was 3.40. Now I have upgraded it to 3.70.1 and am going to test it on Jetpack 4.6 again.

OK. On Jetson Nano, this is my pipeline:


	if (strcmp(cmd_name, "gst_loopback") == 0)
		pipe_proc = "decodebin ! autovideoconvert ! "
			"video/x-raw,format=I420 ! identity drop-allocation=true !"
			"v4l2sink device=/dev/video1 qos=false sync=false";
	else
		pipe_proc = " decodebin ! autovideosink sync=false qos=false";

this is how I am controlling the resolution from the command line. The nano can’t effectively process the 4K streams for object recognition in my tests. You may get better results.

	if (argc > 1 && strcmp("--format", argv[1]) == 0) {
		if (argc > 2 && strcmp("4K", argv[2]) == 0) {
			printf("THETA live video is 4K");
			res = thetauvc_get_stream_ctrl_format_size(devh,
				THETAUVC_MODE_UHD_2997, &ctrl);	
		} else if (argc > 2 && strcmp("2K", argv[2]) == 0) {
			printf("THETA live video is 2K");
			res = thetauvc_get_stream_ctrl_format_size(devh,
				THETAUVC_MODE_FHD_2997, &ctrl);				
		}
		
		else {
			printf("specify video device. --format 4K or --format 2K\n");
			goto exit;
		}
		
	}

Based on Ubuntu 18.04.5 LTS

lsb_release  -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

confirmed with nvv4l2decoder

tested on Jetson Nano. You may need to specify nvv4l2decoder for Xavier as I’ve heard that decodebin may not work.

	if (strcmp(cmd_name, "gst_loopback") == 0)
		// for Jetson Nano
		pipe_proc = "nvv4l2decoder ! autovideoconvert !"
		// pipe_proc = "decodebin ! autovideoconvert ! "
			"video/x-raw,format=I420 ! identity drop-allocation=true !"
			"v4l2sink device=/dev/video1 qos=false sync=false";
	else
		// pipe_proc = " decodebin ! autovideosink sync=false qos=false";
		// tested on Jetson Nano.  Should work on Xavier
		pipe_proc = "nvv4l2decoder ! nv3dsink sync=false qos=false";

starting gst_loopback

Ignoring errors about pixformat. See below. The OP got it to work with the error.

Gstreamer1.0 v4l2sink will not work at all for me. · Issue #137 · umlaeute/v4l2loopback · GitHub

./gst_loopback  --format 2K
Opening in BLOCKING MODE 
libv4l2: error getting pixformat: Invalid argument
Opening in BLOCKING MODE 
start, hit any key to stop
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 

testing OpenCV python script

Note my THETA V is on /dev/video1
tested on Aug 31, 2021 with firmware 3.70.1

python canny.py --video_device 1

Test code

Installed VS Code

To make testing easier, I installed VS Code ARM64 on the Nano. The Jetson is such a cool little device. IMO, it’s much easier to work on than a Raspberry Pi. However, I guess it does cost more than an RPi and likley has more power requirements. The entire Jetson line is great.

image

I changed the codes to

if (strcmp(cmd_name, "gst_loopback") == 0)
		// for Jetson Nano
		pipe_proc = "nvv4l2decoder ! autovideoconvert !"
		// pipe_proc = "decodebin ! autovideoconvert ! "
			"video/x-raw,format=I420 ! identity drop-allocation=true !"
			"v4l2sink device=/dev/video0 qos=false sync=false";
	else
		// pipe_proc = " decodebin ! autovideosink sync=false qos=false";
		// tested on Jetson Nano.  Should work on Xavier
		pipe_proc = "nvv4l2decoder ! nv3dsink sync=false qos=false";

and by running ./gst_loopback I got

./gst_loopback --format 2K
Opening in BLOCKING MODE
ArgusV4L2_Open failed: No such file or directory
Opening in BLOCKING MODE 
libv4l2: error getting pixformat: Invalid argument
Opening in BLOCKING MODE 
start, hit any key to stop
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Error: Internal data stream error.
stop

I googled the ArgusV4L2_Open thing and still have no idea about what it is.
:sweat_smile:

I just tried JetPack 4.4.1, but when I run apt upgrade, I got

Errors were encountered while processing:
 nvidia-l4t-bootloader
E: Sub-process /usr/bin/dpkg returned an error code (1)

in the end.

Has anyone experienced this?

I’m not sure what ArgusV4L2 is myself. However, I think that libargus is the library that the Jetpack Linux4Tegra uses for some cameras.
generic-no-api_r2

Jetson Linux API Reference: Libargus Camera API

You may be able to get additional help on the NVIDIA developer forum. I’m not sure why it wouldn’t be installed with JetPack. The file should be there.

I’ve never seen this before. Did you create a new microSD card?

Yes, I downloaded Jetpack 4.4.1 and cloned it to a new SD card.

I’m downloading JetPack 4.6 now. I’ll try installing it on a new microSD card and see if I can get the streaming working. It may take me a while.

Great! Looking forward to it.

Dear Craig,

I tried using your Canny program in the Jetson Xavier, and I’m still getting the "Internal data stream error".

$ python canny.py --video_device 7
Called with args:
Namespace(video_device=7)
OpenCV version: 4.5.3
Device Number: 7
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (2057) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Internal data stream error.
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (1034) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (597) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_v4l.cpp (890) open VIDEOIO(V4L2:/dev/video7): can't open camera by index
camera open failed

I found this extremely weird. Because even though the pipeline is working with gst_loopback, it is not working on its own in the terminal nor in OpenCV.

I tried different pipelines configurations, but yet nothing seems to be working. Do you have any idea on what could be wrong?


System config:

Jetson Xavier
Ricoh Theta V
Ubuntu 18.04
OpenCV 4.5.3
Python 2.7.17

Now, if I try the following code:

import sys
import cv2

print (cv2.__version__)
print(cv2.__file__)
#print(cv2.getBuildInformation())

cap = cv2.VideoCapture('decodebin ! autovideoconvert ! video/x-raw,format=I420 ! identity drop-allocation=true ! v4l2sink device=/dev/video7  qos=false sync=false', cv2.CAP_GSTREAMER)

if not cap.isOpened() :
        print("capture failed")
        exit()

ret,frame = cap.read()
while ret :
        cv2.imshow('frame',frame)
        ret,frame = cap.read()
        if(cv2.waitKey(1) & 0xFF == ord('q')):
                break;

cap.release()
cv2.destroyAllWindows()

I am getting this error:

/usr/local/lib/python2.7/dist-packages/cv2/python-2.7/cv2.so
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (944) open OpenCV | GStreamer warning: cannot find appsink in manual pipeline
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (597) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
capture failed

If I change cv2.VideoCapture() to:

cap = cv2.VideoCapture('omxh264dec ! gldownload ! glimagesink ! videoconvert n-thread=0 ! video/x-raw,format=I420 ! identity drop-allocation=true ! v4l2sink device=/dev/video7 qos=false sync=false', cv2.CAP_GSTREAMER)

The error is:

/usr/local/lib/python2.7/dist-packages/cv2/python-2.7/cv2.so
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (854) open OpenCV | GStreamer warning: Error opening bin: no property "n-thread" in element "videoconvert0"
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (597) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
capture failed

With:

cap = cv2.VideoCapture('v4l2src device=/dev/video7 ! video/x-raw, width=640, height=480 ! videoconvert ! video/x-raw,format=BGR ! appsink', cv2.CAP_GSTREAMER)

Error is:

/usr/local/lib/python2.7/dist-packages/cv2/python-2.7/cv2.so
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (2057) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Internal data stream error.
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (1034) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/spacer/Downloads/opencv-4.5.3/modules/videoio/src/cap_gstreamer.cpp (597) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
capture failed