Live Streaming from Theta V to Jetson Nano or Raspberry Pi

THETA API Usage
,
5

And here’s a step by step guide on what barely worked but is not recommended and doesn’t solve my problem. It is based off of “Low latency (0.4s) H.264 livestreaming from Theta V WiFi to html5 clients with RTSP plug-in, ffmpeg and Janus Gateway on raspberry Pi”. The problem is not latency but the frames captured (2-3 per second max) and system crashing all the time.

Part A: Configuring Jetson Nano as a wireless access point:

  1. Connect your USB wifi dongle. Type lsusb to see if it has been detected. You should see something like: Bus 001 Device 003: ID 148f:5372 Ralink Technology, Corp. RT5372 Wireless Adapter. Type sudo lshw -C network to check if your device is detected as wlan0. Also type iwconfig and you should find wlan0
  2. sudo apt install util-linux bash procps hostapd iproute2 iw haveged net-tools dnsmasq iptables
  3. sudo nano /etc/dhcp/dhcpd.conf
  4. Find the following section and comment it out by placing a hashtag at the beginning of the line:

option domain-name “example.org”;

option domain-name-servers ns1.example.org, ns2.example.org;

  1. Next un-comment the word authoritative (remove hashtag):
  2. Add the following to the end of line:
subnet 192.168.10.0 netmask 255.255.255.0 {

 range 192.168.10.10 192.168.10.20;

 option broadcast-address 192.168.10.255;

 option routers 192.168.10.1;

 default-lease-time 600;

 max-lease-time 7200;

 option domain-name "local-network";

 option domain-name-servers 8.8.8.8, 8.8.4.4;

}
  1. CTRL+X -> y -> Enter to exit Nano, save the file and exit.
  2. sudo nano /etc/default/isc-dhcp-server
  3. Change INTERFACESV4=” “ to INTERFACESv4=“wlan0”
  4. Save and exit Nano
  5. sudo nano /etc/network/interfaces. Erase everything in that file and type the following:
auto eth0 

auto lo

iface lo inet loopback

iface eth0 inet dhcp

allow-hotplug wlan0

iface wlan0 inet static

 address 192.168.10.1

 netmask 255.255.255.0
  1. Save and exit Nano
  2. sudo nano /etc/hostapd/hostapd.conf. Erase everything in that file and type the following:
interface=wlan0

ssid=nano

channel=6

macaddr_acl=0

auth_algs=1

ignore_broadcast_ssid=0

wpa=3

wpa_passphrase=predator

wpa_key_mgmt=WPA-PSK

wpa_pairwise=TKIP

rsn_pairwise=CCMP
ieee80211n=1

hw_mode=g

device_name=RT5372

manufacturer=Ralink

Note that you can put your own SSID name and wpa_passphrase. Here it’s set as “nano” and “predator”. Device name and manufacturer should come from lsusb and lshw (vendor) outputs earlier.

  1. Save and exit nano
  2. sudo nano /etc/sysctl.conf
  3. Scroll down to the last line of the file and add the line:

net.ipv4.ip_forward=1

  1. Save and exit nano
  2. sudo nano /etc/default/hostapd . change the line: #DAEMON_CONF="" to:
DAEMON_CONF="/etc/hostapd/hostapd.conf".

Save and exit nano.

  1. sudo sh -c “echo 1 > /proc/sys/net/ipv4/ip_forward”
  2. sudo ifup wlan0
  3. sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  4. sudo iptables -A FORWARD -i eth0 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i wlan0 -o eth0 -j ACCEPT
  1. sudo service isc-dhcp-server start

sudo service hostapd start

  1. If you get an error message when starting hostapd, do the following:
sudo nmcli radio wifi off

sudo rfkill unblock wlan

sudo ifconfig wlan0 10.15.0.1/24 up

sleep 1

sudo service isc-dhcp-server restart

sudo service hostapd restart

sudo service isc-dhcp-server start

sudo service hostapd start
  1. Enable the services on reboot

sudo update-rc.d hostapd enable

sudo update-rc.d isc-dhcp-server enable

  1. sudo sh -c “iptables-save > /etc/iptables.ipv4.nat”
  2. Add the following to the end of the file /etc/network/interfaces to restore the configuration when the network interface comes up:

up iptables-restore < /etc/iptables.ipv4.nat

  1. sudo reboot then check if everything works. If everything is fine you should see that your newly made wifi named “nano” pop up on other devices. Make sure you change the wifi access point to this one on Ricoh Theta V using the guide shown previously.
  2. You can use nmap to find ip of the camera connected to the network. Sudo apt-get install nmap, then nmap -sP 192.168.10.*
  3. You can also use iw dev wlan0 station dump to see your camera’s MAC address connected to wlan0 as well as other information.

Part B: Configuring Jetson Nano as a RTSP/RTP streaming sever:

  1. Follow all steps for configuring Jetson Nano as wireless access point
  2. sudo apt-get install libmicrohttpd-dev libjansson-dev libnice-dev libssl-dev libsrtp-dev libsrtp2-dev libsrtp1-dev libsofia-sip-ua-dev libglib2.0-dev libopus-dev libcurl4-openssl-dev liblua5.3-dev libogg-dev libini-config-dev libcollection-dev pkg-config gengetopt libtool automake dh-autoreconf gstreamer1.0-rtsp gsteamer1.0-opencv gstreamer1.0-tools nginx nginx-full janus janus-demos janus-tools janus-dev
  3. Add the following lines to the file /etc/janus/janus.plugin.streaming.cfg
[theta-rpi]

type = rtp

id = 1

description = Theta

audio = false

video = true

videoport = 8080

videopt = 96

videortpmap = H264/90000

videofmtp = "profile-level-id=42e028\;packetization-mode=1
  1. In the same directory, open janus.cfg and set debug level to 0.
  2. sudo cp -r /usr/share/janus/demos/ /usr/share/nginx/html/ (Copy contents of janus demos to nginx html
  3. sudo service nginx start
  4. sudo service janus start
  5. Start the camera and enable the device webapi plugin (https://www.gclue.io/theta/en/). From the web interface, activate live streaming and copy the rtsp url
  6. ffmpeg -i rtsp://192.168.10.11:8086 -an -c:v copy -flags global_header -bsf dump_extra -f rtp rtp://192.168.10.1:8080

Here, the ip after rtsp is obtained for camera, the one with rtp is for host device (nano in this case)

  1. If all is right, you should see frames being captured. How to open them? I couldn’t get janus streaming.html working but i did manage to capture rtsp frames to python.

Part C: Capturing RTSP frames to openCV:

import os

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "dummy"

import cv2

vcap = cv2.VideoCapture("rtsp://192.168.10.11:8086/out.h264")

while(1):

ret, frame = vcap.read()

cv2.imshow('VIDEO',frame)

cv2.waitkey(1)

Part D: Alternative plugin usage:

  1. Install RTSP plugin: https://pluginstore.theta360.com/plugins/com.sciencearts.rtspstreaming/ Instructions on how to install plug in are given there.
  2. Python Code:
import os

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "dummy"

import cv2

vcap = cv2.VideoCapture("rtsp://192.168.10.11:8554/live?resolution=640X320")

while(1):

    ret, frame = vcap.read()

    cv2.imshow('VIDEO',frame)

    cv2.waitKey(1)