Low latency (0.4s) H.264 livestreaming from Theta V WiFi to html5 clients with RTSP plug-in, ffmpeg and Janus Gateway on raspberry Pi

[Hugues]

Hello,

After many tries on Theta V, I achieved to get a H.264 livestreaming on a Raspberry Pi for multiples html5 clients with a 0.4s latency.
It uses almost all realtime protocols available and powerfull softwares just to bypass USB livestreaming witch is not available on linux !

How it works:

• Theta V with RTSP plug-in connected in client mode (should work in AP mode too, 5GHz WiFi for better bandwith). RTSP with Device WebAPI Plug-in also works.
• ffmpeg (running on a Raspberry Pi) gets the RTSP stream and send it in rtp to Janus Gateway.
• Janus Gateway (running on a Raspberry Pi) transforms rtp stream and distributes it to html5 clients (tested on both FireFox and Chrome) with webRTC.

How to do it:

• Install Janus Gateway on the Pi (tutorial : https://www.rs-online.com/designspark/building-a-raspberry-pi-2-webrtc-camera). The gstreamer part is not needed and you can use any webserver.
  My janus.plugin.streaming.jcfg:

theta-rpi: {
    type = "rtp"
    id = 1
    description = "Theta"
    audio = false
    video = true
    videoport = 8080
    videopt = 96
    videortpmap = "H264/90000"
    videofmtp = "profile-level-id=42e028;packetization-mode=1"
} 

Debug level to 0 in janus.jcfg for better Cpu usage

• Install fmpeg on the Pi (already installed with full raspbian) and run it:

ffmpeg -i rtsp://192.168.1.48:8554/live?resolution=1920x960 -an -c:v copy -flags global_header -bsf dump_extra -f rtp rtp://192.168.1.28:8080

192.168.1.48 is the Theta IP address,192.168.1.28 is the Raspberry Pi address
You can use vlc instead but it add latency and when i change the buffer size (1000ms by defauft) it is not stable.

vlc -vvv rtsp://192.168.1.48:8554/live?resolution=3840x1920 --network-caching=1000 --sout “#rtp{dst=192.168.1.28,port=8080}”

• Install RTSP plug-in and Theta V, start Theta V, wait for Wifi connection, start the plug-in

Pros:

• It works !
• low latency => 330 ms
  My solution to test latency: screen capture with chronometer
  
  
• Multiple clients
• low cpu usage (it works on a Pi Zero W), example in 1920x960 with 2 clients
  
  
• 4K available
• Endless streaming if theta V is connected with external power (tested in fullHD)
• Bonus : after the plug-in is started, you can connect theta in USB and download files from it at the same time

Cons:

• Low bitrate, 30fps/2Mbps in 1920x960, 18 fps/3Mbps in 4K
  => more powerfull Pi needed for 4K
• Equirectangular format (need three.js for better view)
  => use RTSP with Device WebAPI Plug-in for a standard view or VR view
• Some freezing

Next:

• Add audio (not needed for my goal)
• Rebuild my three.js implementation (it consumes lot of cpu on the client side)
• See if gophoto2 works (I need powerOff function only available in USB mode) and how to start the plug-in even if the theta is connected in USB (for now i need to disconnect Theta from USB and start the plug-in with the wifi API)
• Use ffmpeg to get both streaming and record on the Pi (easy task)
• Help needed for a new plug-in with all Theta functionnalities and better bitrate

All this work to see wonderfull images from space, NO just sewer pipes

Hugues

[codetricity]

Hugues, this is fantastic. Thanks for sharing it. Your project is really advancing. Are you selling your product globally? It seems like there’s a need for a low-cost way to assess different types of pipelines and sewer systems.

Under cons, you list RTSP. I don’t know too much about RTSP and I’m curious as to the downside. Why is it a con?

You also indicate that you need to disconnect the THETA from USB and start the plug-in with the Wi-Fi API.

Just an FYI, I’ve seen a demo of a usermode Linux streaming driver that allows the THETA to livestream on Linux. I believe the developer is planning to release it as open source in the next few months. When the driver is released, it could be another possible strategy for you.

Update: Nov 13, 2019

@Hugues, can you confirm that this diagram is accurate?

[Moriska32]

Hello! Can you answer some question? I want to get stream video 360 with nvidia jetson nano in 4k. Can i do it with ffmpeg by your solution? And with ffmeg to stream video on my server?

[craig]

If you have a Jetson, you should give it a try. The note from @Hugues indicates that a more powerful Raspberry Pi was needed for 4K. It’s possible that the Nvidia Jetson has enough power.

I’m not sure if anyone has tried this solution with a Jetson.

[Moriska32]

I have Jetson, but don’t have Theta V. I want buy it, but before to buy , i want to know, it solution works on any linux OS, i think it must works, becouse Rasberry Pi i can install on Jetson.

[craig]

The solution in this topic using motionJPEG to connect to the RaspberryPi with Wi-Fi. The connection between the RPi and the camera is Wi-Fi, not a USB cable.

At the current time, the THETA V won’t connect to the Jetson with a USB cable as the output of the THETA V is UVC 1.5 and there is no public patch to libuvc to get the THETA to stream UVC1.5 over a USB cable.

[Moriska32]

Did I understand correctly that you can get video stream from any device via wifi via ffmpeg?

[craig]

The THETA V can serve as a Wi-Fi access point and it can also connect to a router.

Once connected, it can stream MotionJPEG video to another device that can handle HTTP.

I don’t have experience with ffmpeg.

Be aware:

1. MotionJPEG live streaming is primarily useful for surveillance, inspection, and checking things out with a drone
2. You may not get satisfactory experiences if you are trying for telepresence or entertainment. The framerate and resolution may be lower than you want

If you are trying to use the THETA V for telepresence, most people connect the THETA V to another computer with a USB cable. This only works on Windows/Mac.

Can you write down your intended use of the THETA V live stream on the Jetson Nano? It may not work for your application if you are trying to do things like object detection or measurement.

The MotionJPEG stream is likely going to be of lower resolution and lower framerate compared to the H.264 video stream from the USB cable.

The USB cable live stream won’t work with Linux at the moment.

[Moriska32]

Thanks, when i will try streaming video on Theta V by this metod, i post here my results.

[craig]

You may want to buy the camera from a place that has a 30 day return policy in case it doesn’t function as you expect. The MotionJPEG streaming technique is not the “main” streaming technique used to stream to YouTube/Facebook. It has limitations. The framerate may be lower than you need for your application.

[carlo_sgambato]

Good evening, I would like to install janus on my raspberry pi4 to set up a small webrtc streaming server. could you point me to a guide?
thanks

[craig]

hughes posted this link which covers the janus installation and setup.

Building a Raspberry Pi 2 WebRTC camera

[carlo_sgambato]

thanks, but i have a camlink capture card does it change anything?