Live Streaming over USB on Ubuntu and Linux, NVIDIA Jetson

craig · September 27, 2020, 2:39pm

Great news about the progress.

Regarding RTSP or a way to get the stream to another server, I believe other people are working on the same problem, but I do not have an answer at the moment. For example, @Yu_You was asking about this. I believe he moved from the RTSP plug-in to the USB cable with libuvc (the technique in this thread). I do not know if he was then able to use something like gst-rtsp-server to get the stream to another computer.

@Hugues is quick far along using the Janus Gateway to get rtp output on IP networks. I don’t know how busy he is, but if your firm is working on a big project, it might be worthwhile to consider trying to hire him as a consultant. He’s using his FOX SEWER ROVER in production and he’s been freely contributing his knowledge to this group.

Another interesting transmission project is the Lockheed Martin Amelia Drone by @Jake_Kenin, @sjm7783 and others.

There is more information on their project here.

As some of them were in undergraduate school before COVID-19 shut down their project, it might be possible to hire some of the team members as interns.

I’m trying to connect people in parallel to sharing whatever I know because there has been a surge of activity around live streaming and USB camera control, likely due to a maturation of technologies and the increase in demand for remote surveillance and analysis.

There seems like many people are working on foundational knowledge such as transmitting data to a remote server running gscam.

I don’t have ros installed.

Are you saying that you can’t get gscam working on the same computer that the THETA is plugged into with a USB cable.? The documentation examples are using /dev/video*. What is the error message?

Are you using ROS Noetic (Ubuntu 20.04) or ROS Melodic (Ubuntu 18.04) or something else? I’m likely gong to install ROS at some point in the future and test basic camera functionality.

zdydek · September 28, 2020, 2:19am

Craig–

Thanks for the information. After trying various approaches, I’ve settled on a setup that I’m happy with. I’ll document it here for posterity.

Firstly, I modified the pipe_proc line in the libuvc-theta-sample program to put the stream into a udpsink:

pipe_proc = " rtph264pay name=pay0 pt=96 ! udpsink host=127.0.0.1 port=5000 sync=false ";

I then run the test-launch program from the gst-rtsp-server server project with the following pipeline:

./test-launch "( udpsrc port=5000 ! application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264 ! rtph264depay ! h264parse ! rtph264pay name=pay0 pt=96 )"

I tried various other methods of connecting these two gstreamer processes, including shmsink/shmsrc, but ultimately this one worked the best. At some point in the future I may combine the gst_view and test-launch functionality into one executable and do away with some of the needless complexity.

Finally, I used gscam to bring the stream into ROS with the following command:

GSCAM_CONFIG="rtspsrc location=rtspt://10.0.16.1:8554/test latency=400 drop-on-latency=true ! application/x-rtp, encoding-name=H264 ! rtph264depay ! decodebin ! queue ! videoconvert"  roslaunch gscam_nodelet.launch

Note the “rtspt” protocol, it is not a typo. It forces the RTSP connection to go over TCP. When I used UDP there were too many artifacts and corrupted frames

I actually run this last command on a separate machine, just because of my particular network topology. It could be run on the same machine. In fact, it might be necessary to do so because gscam doesn’t seem to handle udp streams well. I also tried using OpenCV VideoCapture to get the data into ros, but that had a couple issues. There are two APIs for VideoCapture that seemed appropriate: Gstreamer and FFmpeg. It turns out that the OpenCV version packaged with ROS is not built with Gstreamer support, so you would have to build OpenCV yourself to use it. For FFmpeg, the version of OpenCV packaged with ROS melodic is 3.2, which is missing a fairly critical change here: https://github.com/opencv/opencv/pull/9292 that allows you to set FFmpeg capture options in an environment variable. I got both of these working by upgrading OpenCV to version 3.4.9 and building from source, but Gstreamer had a low framerate (~5fps) and FFmpeg had a lot of corruption and dropped frames (maybe it was stuck using UDP?). So, I decided to stick with gscam for now.

The latency value of 400 worked for me, but should be tuned depending on your network.

Hope this helps you or someone else wanting to use this camera in ros. So far it looks great and should be perfect for my application. The only negative for me is that I can’t control the power state and streaming state programmatically, unless I missed something in the USB API section. For now I’ve disabled sleep so I only have to turn it on once my robot is up and turn it off when done.

craig · September 28, 2020, 3:25pm

Love it!

Thanks so much for sharing this.

See this video, I think is what you want.

More information is here is the camera section.

https://theta360.guide/special/linuxstreaming/

There is sample code for Jetson Nano as well in that document.

If this is what you are looking for, feel free to ask more questions.

Note that I haven’t tested isolating individual USB ports on the Jetson. I don’t know if it is possible to just reset the USB port that the THETA is attached to. On the Raspberry Pi, all the USB ports are reset when the camera is restarted. If you have other devices attached the USB port of the Xavier that you need, please adjust accordingly. This is not an approved part of the API. If this is something you need for production, we’ll try and test it more.

If your application can live with sleep and wake, it is better as it is part of the official API.

The API supports switching from image to live streaming and to video file.

Note if you need to save video to file, there is a third parameter you must set. It’s documented in on the guide and also on the USB thread of this forum. I think it is something like adding ,0,0,1 to the
end of the hex value for video to file. If you have problems, we can check it.

Joe · May 26, 2018, 3:25pm

I have live streaming working on Mac and everything is great - but I am trying it on ubuntu and its not working. At all. It doesn’t recognize that the camera is present.

Is there a way to make the Ricoh theta v work on ubuntu?

craig · September 30, 2020, 12:15am

This was an old question that I merged with this topic to help people using the search feature of this forum. yes, it’s possible. Please review the topic at the topic and ask more question if you have any problems. Hope you’re still using the camera.

Jaap · October 1, 2020, 12:18pm

Hi I made a few small modifications to a github entry to transform equirectangular images/frames to perspective images. This can really help when using existing deep learning models or when creating new models based on regular (non 360 camera footage. The Github link can be found here.

Original Images

Transformed image

I will try some of this out on some Mobilenet V2 models later.

Community Ask: Functionalize this in a way that is fast and uses Nvidea GPU on Jetson or Xavier AGX.

craig · October 1, 2020, 1:34pm

Nice. This is great. Thanks for sharing it.

I tested it with Python3 on x86 using the standard opencv shipped with Ubuntu 20.04 and Z1 jpeg images. Works great.

sudo apt install python3-opencv

The only sad thing I noticed is that all the test pictures in my camera are of me sitting in front of a computer.

Have you tried it on a video stream without applying Mobilenet v2 analysis? I’m wondering how fast the Equirec2Perspec can handle the frames.

I’m still trying to learn more about this new frontier (for me) of GPU acceleration on the Jetson.

Just to clarify the challenge you’re proposing, do you mean to test OpenCV on NVIDIA Jetson with CUDA acceleration?

As my knowledge is weak, I do not know if I need to modify the Python code to load the CV portions into the GPU or if compiling OpenCV with CUDA support somehow automatically does this for me.

Most of the documentation I can find just focuses on compiling CUDA support into OpenCV

I’ve compiled OpenCV from source with CUDA support on the Nvidia Jetson, but I’m not sure if I need to do stuff like cv.cuda…

If you have Equirec2Perspec working on a stream, we can also just test it and see what the latency is. If it’s too slow, then we can try and modify the code with the cv.cuda names.

skimura · October 2, 2020, 12:32pm

Thanks to this topic, I was able to run gst_viewer on my Ubuntu laptop and view streaming video of THETA V.

Now,I have a question about THETA video streaming over USB.
This may seem like an odd question, but I would like to ask someone to give me an advice if possible.

I’m trying to view streaming video on ffplay with gst_loopback,v4l2loopback and ffmpeg as well as gst_viewer.
You might think "Why is this guy trying to use ffplay even though gst_viewer is available?”. I know, but it’ll be a long story.So I’ll skip it now.

Anyway, I connected THETA and ran gst_loopback and ran the following command.
("/dev/video1" is a device file created by v4l2loopback.)

ffmpeg -i /dev/video1 -f matroska - | ffplay -

However, as shown in the screenshot, bit rate indicates N/A and no streaming image was displayed.

I changed /dev/video1 to /dev/video0 (the web-camera built into the laptop) and ran the same command and confirmed that the video from the web-camera was correctly displayed.
To further confirm, I ran the following command and saved a 10 second video.

ffmpeg -i /dev/video1 -f matroska output.mp4

The length of the output.mp4 was 0 seconds and only one image was stored.

gst_viewer is working fine, so I think I’m missing some settings or making a mistake on the v4l2loopback, ffmpeg side, but I couldn’t find a good solution.
Does anyone have an advice on this?

environment:
camera:THETA V
OS:Ubuntu 18.04 LTS 64bit
CPU:i7-9750H
RAM:16GB

craig · October 2, 2020, 9:59pm

I do not have a solution, but I can replicate the problem. I will keep trying.

I have the same problem that you do.

The only thing I got working is that ffplay works with THETA on /dev/video0

$ ffplay -f v4l2 /dev/video0
ffplay version 3.4.8-0ubuntu0.2 Copyright (c) 2003-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100

craig · October 3, 2020, 3:17am

I put it into a loop to get the frames from RICOH THETA Z1 to see if it was using the GPU and I don’t think it is.

This screen grab is coming over an X forwarding session to another computer, so it may be faster on a monitor plugged into the Jetson. If I increase the window size, the latency gets worse. The interesting challenge is to get cv2.cuda.remap working, which I don’t have working.

equi_test

import cv2
import os
import Equirec2Perspec2 as E2P

print(f"OpenCV version {cv2.__version__}")
cap = cv2.VideoCapture(0)

# Check if the webcam is opened correctly
if not cap.isOpened():
    raise IOError("Cannot open webcam")

while True:
    ret, frame = cap.read()
    #print(frame.shape)    
    equ = E2P.Equirectangular(frame)
    frame = equ.GetPerspective(120, 180, -15, 400, 400)
    cv2.imshow('Input', frame)

    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

in Equirec2Perspec2.py

class Equirectangular:
    def __init__(self, img):
        self._img = img
        [self._height, self._width, _] = img.shape

There’s a guy at the link below trying to use cv2.cuda.remap for the same thing we are.

In the comments, it ends with a happy “it’s working” with a plethora of exclamation points.

I’m not sure what this cv2.fisheye… is doing

map1, map2 = cv2.fisheye.initUndistortRectifyMap(K, D, np.eye(3), K, DIM, cv2.CV_16SC2)#

But, other than that, loading the map looks doable.

        #map1 = map1.astype(np.float32)
        #map2 = map2.astype(np.float32)
        #map1 = cv2.cuda_GpuMat(map1)
        #map2 = cv2.cuda_GpuMat(map2)

then at some point, he’s able to use cv2.cuda.resize with hopefully faster processing…

        #resized = cv2.cuda.resize(img2, (new_w, new_h), interpolation=cv2.INTER_LINEAR)

Another example of using cv.cuda.remap by the same guy, with another “it’s working” at the end

Some friendly guy posted this, which appears to work, according to the original poster.

cuDst = cv.cuda_GpuMat(cuMat.size(),cuMat.type())
cv.cuda.remap(cuMat,cuMapX,cuMapY,dst=cuDst,interpolation=cv.INTER_LINEAR)

We probably just need to read up on cv.cuda and the GpuMat and maybe we’ll get it too, like the happy guy posting on the OpenCV forums.

Jaap · October 3, 2020, 11:59am

The only sad thing I noticed is that all the test pictures in my camera are of me sitting in front of a computer.

Ha, for me it is my basement, time to clean up. No place to hide stuff in a 360 image

Have you tried it on a video stream without applying Mobilenet v2 analysis? I’m wondering how fast the Equirec2Perspec can handle the frames.

Not yet, doing this on a video stream is my next to do. I will time that process first before applying any NN. My intuition leads me to believe that many NN run fast enough (100 fps + on the AGX Xavier) that a non-linear transformation will form the bottleneck. The good thing is that I don’t need very fast times. Anything around a second would work for me.

Just to clarify the challenge you’re proposing, do you mean to test OpenCV on NVIDIA Jetson with CUDA acceleration?

No Craig, I was referring to this transformation function specifically. I am already worried that this could be a bottleneck in any application. A C++ implementation using some of the Nvidia accelerations could be very helpful.

jothomas · October 5, 2020, 10:05am

Hello @craig, Were you able to implement this solution?

Jaap · October 6, 2020, 11:35am

Got it to work!! it is streaming and correcting in realtime. Let me play around with the code a little.

pers_1

I can even create a panorama from perspective corrected images.

import cv2
import os
import Equirec2Perspec2 as E2P
import numpy as np

print(f"OpenCV version {cv2.__version__}")
cap = cv2.VideoCapture(1)

# Check if the webcam is opened correctly
if not cap.isOpened():
    raise IOError("Cannot open webcam")

while True:
    ret, frame = cap.read()
    #print(frame.shape)
    equ = E2P.Equirectangular(frame)

    frame1 = equ.GetPerspective(90, 0, 0, 400, 400)
    frame2 = equ.GetPerspective(90, 90, 0, 400, 400)
    frame3 = equ.GetPerspective(90, 180, 0, 400, 400)
    frame4 = equ.GetPerspective(90, 270, 0, 400, 400)

    hor_concat_top = np.concatenate((frame1, frame2), axis=1)
    hor_concat_bottom = np.concatenate((frame3, frame4), axis=1)

    all_four_frames = np.concatenate((hor_concat_top, hor_concat_bottom), axis=1)

    cv2.imshow('Input', all_four_frames)

    c = cv2.waitKey(1)
    if c == 27:
        break

cap.release()
cv2.destroyAllWindows()

-Jaap

craig · October 6, 2020, 1:02pm

Congratulations and great work.

Are you next going to experiment with some type of object detection on the frames?

The technique would also be useful for human analysis. Selecting portions for humans to inspect.

Sorry, I haven’t tried yet.

The primary algorithm looks like it is feasible to convert to cv2.cuda.remap, but I believe, we need to use cv2.cuda_GpuMat() on each frame and then upload the frame to the GPU.

persp = cv2.remap(self._img, lon.astype(np.float32), lat.astype(np.float32), cv2.INTER_CUBIC, borderMode=cv2.BORDER_WRAP)

This is quite an interesting challenge as I think we can get some big gains on a wide range of frame processing with little effort. We just need to figure the correct sequence of steps for a few common methods such as remap, resize, cvtColor.

jcasman · October 6, 2020, 1:39pm

Hey, nice going, Jaap!

LukeLu · October 15, 2020, 7:38pm

Dear Ricoh Theta V friends,

We got internal data stream error when using gst_loopback even a dummy /dev/video* is created and is assigned by v4l2loopback.
By following this nice tutorial, both ptpcam and gst_viewer works well.

But we stuck on gst_loopback, it outputs below error:

start, hit any key to stop
Error: Internal data stream error.
stop

Seems like we are very close, one step to get it working, were wondering if any one got the same issue.

Our desktop testing environment is Ubuntu 18.04 LTS, with Linux 4.15.0-118-generic, x86-64
Camera is Theta V with the firmware 3.50.1

Best,
– Luke

craig · October 15, 2020, 8:17pm

when you do lsmod, can you see the v4l2loopback module?

$ sudo modprobe v4l2loopback
[sudo] password for craig: 
$ lsmod
Module                  Size  Used by
v4l2loopback           40960  0
btrfs                1253376  0
...

Do you have more than one camera on your system? If not, did you modify the device to /dev/video0 on this line?

On x86, change the line 190 as follows:

if (strcmp(cmd_name, "gst_loopback") == 0)
    pipe_proc = "decodebin ! autovideoconvert ! "
        "video/x-raw,format=I420 ! identity drop-allocation=true !"
        "v4l2sink device=/dev/video0 qos=false sync=false";

Make sure qos=false

If you moved the camera around to different systems. Reboot the camera. Hold the power down for about 15 seconds and then reboot it while it is plugged it. I sometimes have problems if I have the camera plugged into one system and then move it.

Please confirm and report back with additional errors. It should work on your system.

LukeLu · October 16, 2020, 3:43am

Craig, thanks for your reply. We have double checked lsmod, /dev/video1 (we have one existing webcam), added qos=false and reboot, and tried exclusive_caps=1 or 0. But internal data stream error is still there. We have two linux systems getting the same error.

As far as we know, the error message is from v4l2loopback, a most related discussion and a potential solution was discussed below:

Before digging into above, we are getting one more theta V to double check it’s not caused by the specific camera.

Will keep you posted.
Best,
– Luke

craig · October 16, 2020, 12:48pm

What is the graphics card and driver you are using?

You can get some information with nvidia-smi

It will look something like this:

Note that the reason is says, “Off” next to the GeForce GTX 950 is because the “Off” refers to Persistence-M (row above). It does not indicate if the GPU is enabled or not. In this case, the driver is the proprietary Nvidia driver 450.66.

The v4l2loopback does not work on all systems.

There is a small note on the README.md for libuvc-theta-sample with potentially big meaning.

If you have an nvidia graphics card, you may need the nvidia plug-in for gstreamer, which I think is in the “plug-ins bad” group, but appears to work. I got this information from someone that is more knowledgeable about gstreamer than I am.

In my case, I don’t really understand gstreamer, so I installed everything.

https://gstreamer.freedesktop.org/documentation/installing/on-linux.html?gi-language=c

apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

You may not need the step above. You can likely just install the individual plug-in categories.

There may still be a graphics card problem.

You can try running the X.org driver, nouveau.

Once you switch to nouveau, you can verify the driver with:

$ glxinfo -B
name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: nouveau (0x10de)
    Device: NV126 (0x1402)
    Version: 20.0.8
    Accelerated: yes
    Video memory: 2024MB

or sudo lshw -c video

Note that I normally use the nvidia driver. I switched over to nouveau after I saw your post to try and test it. I did basic testing with the nouveau driver and the THETA. It appears to run about the same.

snafu666 · October 17, 2020, 12:48pm

i get a slightly different result on my nvidia NX and generic ubuntu 18.04:

nvidia@nx:~/build/libuvc-theta/build$ ./example
UVC initialized
uvc_find_device: No such device (-4)
UVC exited

also note that i do not see any /dev/video* devices but lsusb shows:

nvidia@nx:~/build/libuvc-theta/build$ lsusb
Bus 002 Device 002: ID 0bda:0489 Realtek Semiconductor Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13d3:3549 IMC Networks
Bus 001 Device 005: ID 0c45:7403 Microdia Foot Switch
Bus 001 Device 004: ID 1a40:0101 Terminus Technology Inc. Hub
Bus 001 Device 006: ID 05ca:0368 Ricoh Co., Ltd
Bus 001 Device 002: ID 0bda:5489 Realtek Semiconductor Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub