Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

11112 · July 29, 2020, 1:28pm

i get the gstreamer-app-1.0 ,and try by https://github.com/ricohapi/libuvc-theta-sample.git;

11112 · July 29, 2020, 1:28pm

how to get gstreamer-app-1.0 on ubuntu? i can not find it.

craig · July 29, 2020, 6:03pm

I believe it is in libgstreamer-plugins-base1.0-dev

https://packages.ubuntu.com/xenial/amd64/libgstreamer-plugins-base1.0-dev/filelist

craig · July 29, 2020, 6:05pm

See the video that is now on this info page of this site.

RICOH THETA360.guide Independent developer community (put your email in and you will see a different page)

Please review the section on libuvc compilation with the appropriate dependencies and patch in the video.

update July 29, 11:30am PDT: I added a small troubleshooting section under the video, including a screen shot of remote git branch usage and package location of gstreamer-app-1.0

11112 · July 30, 2020, 1:44am

thank you ,it is ok,and libusb need above 1.0.9;

craig · July 30, 2020, 2:40am

v0.0.6-theta-uvc released 2 hours ago. I am trying to compile on a Raspberry Pi. I will report back.

11112 · July 30, 2020, 3:00am

thanks,I have try. it is OK。
by the way,Can i get livestreaming by usb on android?

I want try control ThetaV by usb on android.

jothomas · July 30, 2020, 8:42am

Thank you for uploading the video. It works for me now. The issue earlier was that I build the master branch instead of theta_uvc.
branch_master
Also, I would like to ask if it is possible to make the Ricoh camera be detected as a video device. In that way, I can directly use it in ROS, OpenCV, or other robotics/computer vision applications.

craig · July 30, 2020, 8:19pm

Hi, I didn’t write the sample code or the patch, but I’ve seen the THETA Z1 accessed from /dev/video0 on Linux. I’m trying to get this to work myself. I’ll share what I know and let’s work together on a solution that we can post here.

I’ve seen the following work on Linux:

gst-launch-1.0 from command line accessing /dev/video0
Yolov3
openpose

Here’s what I’ve seen work but can’t get to work myself.

Note that omxh264dec is specific to NVIDIA Jetson. x86 needs to use vaapih264dec or msdkh264dec (I think. I don’t have it working)

sudo modprobe v4l2loopback
python3 thetauvc.py | gst-launch-1.0 fdsrc ! h264parse ! queue ! omxh264dec ! videoconvert ! video/x-raw,format=I420 ! queue ! v4l2sink device=/dev/video0 sync=true -vvv

./darknet detector demo cfg/coco.data cfg/yolov2.cfg yolov2.weights

in openpose/build/examples/openpose

openpose.bin --camera 0

I’m now working on an x86 Linux. I was able to build the driver and demo on a Raspberry Pi 3, but I was not able to run the demo. It would hang with no error messages. It might be possible with a Raspberry Pi 4.

I am going through the information for Intel Video and Audio for Linux to try and get a better fundamental understanding.

I don’t know what v4l2loopback does, but I’ve built the kernel module and I’ve installed it.

It did create a video device on my system. The first device is my webcam.

$ ll /dev/video*
crw-rw----+ 1 root video 81, 0 Jul 28 18:21 /dev/video0
crw-rw----+ 1 root video 81, 1 Jul 30 10:21 /dev/video1

This section from the README.md of the v4l2loopback site seems promising, but I don’t have it working at the moment.

the data sent to the v4l2loopback device can then be read by any v4l2-capable application.

If you get something working, please post. It looks super-promising, but I can’t access it from a /dev/video at the moment.

jothomas · July 31, 2020, 7:59am

Hi, thanks for the reply. I will take a look into it and see if I can make it work.

Jaap · August 6, 2020, 10:32pm

First of all, great work everyone, I am loving these forums.

I have a Raspberry pi 4, a Jetson Nano and a Jetson Xavier
I would like to stream vidio from my Ricoh V or Z1 to these devices and apply CV, tensorflow, code etc.
IDeally the video would just show up as dev/video0, so frames can be easily grabbed in my Python CV enviroment. I can do this in Windows, however, the max resolution seems to be 1920 - 1080. I cannot do it at all on these devices. I got very exited to see the anouncement for the linux driver on July 29th. I would like to organize notes here and help but together an easy-to-follow, guide on how to get this to work. I think this would open up a world of possibilities for the broader community. So I have the following questions:

Should I even attempt this on my Raspi 4?
Has anyone gotten de Ricoh V/Z1 to show up as dev/video0?
Has anyone been able to grab 4K frames in the python CV enviroment on any OS?

If we can get this don together, I will sepnd time organizing a easy to follow tutorial with approriate credits to anyone who helped.

Jaap · August 7, 2020, 6:29pm

Hi Craig, I got pretty far along on my RPi 4 until the make in gst. I get the following error:

pi@raspberrypi:~/Documents/ricoh/libuvc/libuvc-theta-sample/gst $ pwd
/home/pi/Documents/ricoh/libuvc/libuvc-theta-sample/gst
pi@raspberrypi:~/Documents/ricoh/libuvc/libuvc-theta-sample/gst $ ls
gst_viewer.c  Makefile  thetauvc.c  thetauvc.h
pi@raspberrypi:~/Documents/ricoh/libuvc/libuvc-theta-sample/gst $ make
Package gstreamer-app-1.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `gstreamer-app-1.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gstreamer-app-1.0' found
cc    -c -o gst_viewer.o gst_viewer.c
gst_viewer.c:38:10: fatal error: gst/gst.h: No such file or directory
 #include <gst/gst.h>
          ^~~~~~~~~~~
compilation terminated.
make: *** [<builtin>: gst_viewer.o] Error 1

Jaap · August 7, 2020, 6:53pm

got it to work!!! but the latency on my Raspi is 20 ~seconds.

I installed some of the gstreamer stuff.

 $  sudo apt-get update -y sudo apt-get install -y libgstreamer1.0-dev

  $  sudo apt install gstreamer1.0-plugins-bad gstreamer

  $ sudo apt install libopenal-dev libsndfile1-dev libgstreamer1.0-dev libsixel-dev gstreamer1.0-plugins-bad gstreamer1.0-plugins-base libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev

  $ sudo apt-get install libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-good

jcasman · August 7, 2020, 6:59pm

@Jaap This is great!! I know @codetricity was testing on a Raspberry Pi 3, and I think was having some issues. This looks like good info on gstreamer.

Jaap · August 8, 2020, 12:36am

Thank you, I got the Z1 to stream as well on the Raspi 4. Same delay it seems more than 20 seconds. I will try my Jetson Nano tomorrow. The camera is not showing up under dev/video0. I wonder how I can grab frames and bring them into python to start computer vision applications. Any advise?

jcasman · August 9, 2020, 10:09pm

It may be that the Jetson Nano has more memory or other hardware advantages compared to the Raspberry Pi 4. I don’t know this, I’m just guessing. I’d be interested to hear how it goes with your tests and if there’s an improvement in the latency.

craig · August 10, 2020, 4:24pm

I was unable to use the libuvc-theta and libuc-theta-sample on the Raspberry Pi 3. The platform is too slow. What was the configuration of your Raspberry Pi 4? How much RAM?

I have seen it running on the Jetson Nano with a single 4K live stream with low latency.

I have seen the demo running on Jetson Xavier with 2 4K stream simultaneously.

There was an update to the README about a gst_loopback file and an update to the code.

github.com

ricohapi/libuvc-theta-sample/blob/f8c3caa32bf996b29c741827bd552be605e3e2e2/gst/gst_viewer.c#L190

    
      
          
          
	cmd_name = rindex(argv[0], '/');
          	if (cmd_name == NULL)
          		cmd_name = argv[0];
          	else
          		cmd_name++;
          
          
	if (strcmp(cmd_name, "gst_loopback") == 0)
          		pipe_proc = "decodebin ! autovideoconvert ! "
          			"video/x-raw,format=I420 ! identity drop-allocation=true !"
          			"v4l2sink device=/dev/video1 sync=false";
          	else
          		pipe_proc = " decodebin ! autovideosink sync=false";
          
          
	if (!gst_src_init(&argc, &argv, pipe_proc))
          		return -1;
          
          
	res = uvc_init(&ctx, NULL);
          	if (res != UVC_SUCCESS) {
          		uvc_perror(res, "uvc_init");
          		return res;

gst/gst_loopback

Feed decoded video to the v4l2loopback device so that v4l2-based application can use THETA video without modification.

CAUTION: gst_loopback may not run on all platforms, as decoder to v4l2loopback pipeline configuration is platform dependent,

Examples with VLC and OBS on /dev/video*

I am having issues with the frames stopping, but at least I have the video stream working. I think I need to specifically set the framerate and resolution. I’ll keep looking into this. If anyone has it working smoothly with something like OpenCV, please post your settings.

running with VLC on /dev/video2

running with OBS

update 2018 Aug 17

I heard from the developer that it may be easier to get the v4l2loopback working with an iGPU or with an NVIDIA Jetson. These are the two things he’s tested. I just ordered a Nano to test the loopback with the Nano.

I’m also trying to organize information on using the driver. I’ll put some articles that I’m looking at:

craig · August 19, 2020, 6:10pm

Online Meetup to Discuss Linux Live Streaming

Sept 1, 10am PDT

Signup

https://theta360developers.github.io/linux-streaming-meetup/

Agenda

Introduce Oppkey, .guide, THETA camera
THETA Live Streaming in General
THETA Live Streaming on Linux
- USB cable
- Wi-Fi RTSP
- Wi-Fi motion JPEG
libuvc explanation
gst-streamer demo (already built) and explanation
v4l2
v4l2loopback and /dev/video*
Current status problems and welcome contributions
- Does not work on Raspberry Pi 3. Raspberry Pi 4 shows 20 second latency
- Frames hang after about 100 to 200 frames on /dev/video*

Signup Now

jcasman · August 19, 2020, 6:32pm

I’ll be running the online meetup and will be presenting both a good overview of the tools and techniques immediately available, plus digging down into specifics around the newly available libuvc library and what that means for THETA developers.

If you’re already working with live streaming or you’re just interested in finding out what’s possible, this will be a great learning opportunity.

Also, we’ll provide ample opportunity to ask questions and talk about your current setup, if you’re interested in sharing. Let’s all join in and help grow the community!

Signup Now!

Screen Shot 2020-08-19 at 11.27.11 AM

Juantonto · August 20, 2020, 12:04am

GREAT!
signed up just now.

Live Streaming in 360 is so exciting.

Regards,
Toyo