Live Streaming from Theta V to Jetson Nano or Raspberry Pi

[Swapnil_Sayan_Saha]

My goal is to view 360 degree livestreams from Theta V to Rpi or Jetson Nano. This is what I have done so far:

1. RICOH THETA API Over USB Cable - (Z1, V, S, SC models), Using the THETA API Over USB, https://api.ricoh/docs/theta-usb-api/, https://dzone.com/articles/controlling-iot-cameras-with-libptp, https://github.com/jim-easterbrook/python-gphoto2, https://www2.theta360.guide/doc/article/4 and Live Streaming over USB on Ubuntu Followed the guides here and is now able to control the Theta from Jetson Nano over USB using PTP commands as well as commands from libgphoto2 and its python bindings. However, I am aware that native USB live-stream is not possible on Arm64 Linux architectures and that is the case I have faced too. Since video is made up of images, I wanted to capture multiple frames/images simultaneously using PTP commands and transfer them to computer but it’s too slow.
2. https://support.theta360.com/uk/manual/v/content/streaming/streaming_02.html Followed this guide and managed to view HQ livestream video via USB on a Windows Machine using Media Player Classic. However, this means I will need to use a Windows System (and work by exporting snapshots from Media Player Classic), which I ain’t keen on doing. I tried doing the same with VLC on Linux but as expected it does not detect the theta as a serial camera.
3. https://www.gclue.io/theta/en/, https://pluginstore.theta360.com/plugins/org.deviceconnect.android.manager/, https://support.theta360.com/uk/manual/v/content/prepare/prepare_08.html Used the device web api plugin to stream to the local network (MJPEG and RTSP). While it sort of works, I cannot see the full preview in the live stream and have to use ROI to navigate, which is a hassle. Plus there are latency issues too.

Is there a solution to the problem?

[craig]

This solution may be useful.

Low latency (0.4s) H.264 livestreaming from Theta V WiFi to html5 clients with RTSP plug-in, ffmpeg and Janus Gateway on raspberry Pi

Hello, After many tries on Theta V, I achieved to get a H.264 livestreaming on a Raspberry Pi for multiples html5 clients with a 0.4s latency. It uses almost all realtime protocols available and powerfull softwares just to bypass USB livestreaming witch is not available on linux ! How it works: Theta V with RTSP plug-in connected in client mode (should work in AP mode too, 5GHz WiFi for better bandwith). RTSP with Device WebAPI Plug-in also works. ffmpeg (running on a Raspberry Pi) gets the…

Successful Theta V stream from drone to VR headset 0.25 miles away

Just figured I would drop a little project update here letting people know about the senior design project I worked on that used a Ricoh Theta V. I know @codetricity and others on here would definitely be interested in it, since I have seen people talking about sending Ricoh Theta V video from a drone, but as far as I know noone has demonstrated anything yet. We built a system that would allow a ground operator to view a livestream video from a drone in their VR headset. The system will send th…

[Swapnil_Sayan_Saha]

Here’s a step by step by guide of what works on every platform (Windows, Mac, Linux) (Reference: https://github.com/muimota/thetav_proxy):

1. Install nodejs: https://nodejs.org/en/ For jetson nano: sudo apt-get install nodejs libssl1.0-dev nodejs-dev

2. Write the following code and save it as anyName.js (e.g. thetav.js)

const http = require('http')
const server = http.createServer()


console.log("\nRICOH theta V camera proxy\n2019 @muimota")

server.on('request', (_req,_res) =>
  {

    const data = '{"name": "camera.getLivePreview"}'

    const options = {
      hostname: '192.168.1.1',
      port: 80,
      path: '/osc/commands/execute',
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Content-Length': data.length
      }
    }

    const req = http.request(options, (res) => {

      _res.writeHead(200,
        {'Connection': 'Keep-Alive',
         'Content-Type': 'multipart/x-mixed-replace; boundary="---osclivepreview---"',
         'X-Content-Type-Options':'nosniff',
         'Transfer-Encoding': 'Chunked'})

      res.on('data', (d) => {
        _res.write(d)
      })
    })

    req.on('error', (error) => {
      console.log('Error connecting ricoh camera')
      _res.end('Error connecting ricoh camera')
    })

    req.write(data)
    req.end()

  }
)
server.listen(8000)

3. Optional: only if someone had destroyed the network interface of jetson nano like me after following the guide i posted in the above reply to make the wifi adapter work as a proper wifi adapter again.

• Revert changes to /etc/dhcp/dhcpd.conf
• Revert changes to /etc/default/isc-dhcp-server
• Revert changes to /etc/network/interfaces. Make it:
  source /etc/network/interfaces.d/*
• Remove everything in /etc/hostapd/hostapd.conf
• Revert changes to /etc/sysctl.conf
• Revert changes to /etc/default/hostapd
• Revert startup changes:
  sudo update-rc.d -f hostapd remove
  sudo update-rc.d -f isc-dhcp-server remove
• Revert changes to /etc/iptables.ipv4.nat
• Reboot the system

4. Start Theta V camera with connecting to smartphone via Theta’s Wifi mode (https://support.theta360.com/uk/manual/v/content/prepare/prepare_08.html) << basically the initial steps of how you connected your theta to phone, i.e. change wireless mode such that blue wireless lamp flashes on theta.

5. Connect to theta V’s wifi. For ubuntu, click network manager in ubuntu > click “connect to hidden wifi network” >

6. Connection: New

7. Network name: name that appeared on your smartphone. e.g. THETAYLXXXXXXXX.OSC

8. Wifi security: WPA / WPA 2 Personal. Add password. For me it’s theta’s serial number as usual.

For windows and mac, they should automatically find the network SSID, you just need to put your theta’s serial number in password.

6. in terminal (ubuntu/mac) or powershell (windows): node thetav.js

7. Go to http://127.0.0.1:8000 in Chrome or Firefox. You should see video stream with near-negligible network latency, limited to 65 Mbps (max speed supported by theta V).

8. It’s also possible to use it with gstreamer, which can later be used with cv2.videocapture like i used in my previous reply.
   gst-launch-1.0 -v souphttpsrc location=http://127.0.0.1:8000 ! decodebin ! autovideosink

9. For the method that worked for me (the node js thingy), here’s the python code i use to capture frames:

import os
os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "dummy"
import cv2
import datetime

vcap = cv2.VideoCapture("http://127.0.0.1:800/")
while(1):
    ret,frame = vcap.read()
    cv2.imshow('VIDEO',frame)
    cv2.waitkey(1)

I would say this nodejs solution is pretty good. The fps is consistent at 30 and the latency is 0.3 seconds max.

[Swapnil_Sayan_Saha]

And here’s a step by step guide on what barely worked but is not recommended and doesn’t solve my problem. It is based off of “Low latency (0.4s) H.264 livestreaming from Theta V WiFi to html5 clients with RTSP plug-in, ffmpeg and Janus Gateway on raspberry Pi”. The problem is not latency but the frames captured (2-3 per second max) and system crashing all the time.

Part A: Configuring Jetson Nano as a wireless access point:

1. Connect your USB wifi dongle. Type lsusb to see if it has been detected. You should see something like: Bus 001 Device 003: ID 148f:5372 Ralink Technology, Corp. RT5372 Wireless Adapter. Type sudo lshw -C network to check if your device is detected as wlan0. Also type iwconfig and you should find wlan0
2. sudo apt install util-linux bash procps hostapd iproute2 iw haveged net-tools dnsmasq iptables
3. sudo nano /etc/dhcp/dhcpd.conf
4. Find the following section and comment it out by placing a hashtag at the beginning of the line:

option domain-name “example.org”;

option domain-name-servers ns1.example.org, ns2.example.org;

5. Next un-comment the word authoritative (remove hashtag):
6. Add the following to the end of line:

subnet 192.168.10.0 netmask 255.255.255.0 {

 range 192.168.10.10 192.168.10.20;

 option broadcast-address 192.168.10.255;

 option routers 192.168.10.1;

 default-lease-time 600;

 max-lease-time 7200;

 option domain-name "local-network";

 option domain-name-servers 8.8.8.8, 8.8.4.4;

}

7. CTRL+X -> y -> Enter to exit Nano, save the file and exit.
8. sudo nano /etc/default/isc-dhcp-server
9. Change INTERFACESV4=” “ to INTERFACESv4=“wlan0”
10. Save and exit Nano
11. sudo nano /etc/network/interfaces. Erase everything in that file and type the following:

auto eth0 

auto lo

iface lo inet loopback

iface eth0 inet dhcp

allow-hotplug wlan0

iface wlan0 inet static

 address 192.168.10.1

 netmask 255.255.255.0

12. Save and exit Nano
13. sudo nano /etc/hostapd/hostapd.conf. Erase everything in that file and type the following:

interface=wlan0

ssid=nano

channel=6

macaddr_acl=0

auth_algs=1

ignore_broadcast_ssid=0

wpa=3

wpa_passphrase=predator

wpa_key_mgmt=WPA-PSK

wpa_pairwise=TKIP

rsn_pairwise=CCMP
ieee80211n=1

hw_mode=g

device_name=RT5372

manufacturer=Ralink

Note that you can put your own SSID name and wpa_passphrase. Here it’s set as “nano” and “predator”. Device name and manufacturer should come from lsusb and lshw (vendor) outputs earlier.

14. Save and exit nano
15. sudo nano /etc/sysctl.conf
16. Scroll down to the last line of the file and add the line:

net.ipv4.ip_forward=1

17. Save and exit nano
18. sudo nano /etc/default/hostapd . change the line: #DAEMON_CONF="" to:

DAEMON_CONF="/etc/hostapd/hostapd.conf". 

Save and exit nano.

19. sudo sh -c “echo 1 > /proc/sys/net/ipv4/ip_forward”
20. sudo ifup wlan0
21. sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
22. sudo iptables -A FORWARD -i eth0 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo iptables -A FORWARD -i wlan0 -o eth0 -j ACCEPT

23. sudo service isc-dhcp-server start

sudo service hostapd start

24. If you get an error message when starting hostapd, do the following:

sudo nmcli radio wifi off

sudo rfkill unblock wlan

sudo ifconfig wlan0 10.15.0.1/24 up

sleep 1

sudo service isc-dhcp-server restart

sudo service hostapd restart

sudo service isc-dhcp-server start

sudo service hostapd start

25. Enable the services on reboot

sudo update-rc.d hostapd enable

sudo update-rc.d isc-dhcp-server enable

26. sudo sh -c “iptables-save > /etc/iptables.ipv4.nat”
27. Add the following to the end of the file /etc/network/interfaces to restore the configuration when the network interface comes up:

up iptables-restore < /etc/iptables.ipv4.nat

28. sudo reboot then check if everything works. If everything is fine you should see that your newly made wifi named “nano” pop up on other devices. Make sure you change the wifi access point to this one on Ricoh Theta V using the guide shown previously.
29. You can use nmap to find ip of the camera connected to the network. Sudo apt-get install nmap, then nmap -sP 192.168.10.*
30. You can also use iw dev wlan0 station dump to see your camera’s MAC address connected to wlan0 as well as other information.

Part B: Configuring Jetson Nano as a RTSP/RTP streaming sever:

1. Follow all steps for configuring Jetson Nano as wireless access point
2. sudo apt-get install libmicrohttpd-dev libjansson-dev libnice-dev libssl-dev libsrtp-dev libsrtp2-dev libsrtp1-dev libsofia-sip-ua-dev libglib2.0-dev libopus-dev libcurl4-openssl-dev liblua5.3-dev libogg-dev libini-config-dev libcollection-dev pkg-config gengetopt libtool automake dh-autoreconf gstreamer1.0-rtsp gsteamer1.0-opencv gstreamer1.0-tools nginx nginx-full janus janus-demos janus-tools janus-dev
3. Add the following lines to the file /etc/janus/janus.plugin.streaming.cfg

[theta-rpi]

type = rtp

id = 1

description = Theta

audio = false

video = true

videoport = 8080

videopt = 96

videortpmap = H264/90000

videofmtp = "profile-level-id=42e028\;packetization-mode=1

4. In the same directory, open janus.cfg and set debug level to 0.
5. sudo cp -r /usr/share/janus/demos/ /usr/share/nginx/html/ (Copy contents of janus demos to nginx html
6. sudo service nginx start
7. sudo service janus start
8. Start the camera and enable the device webapi plugin (https://www.gclue.io/theta/en/). From the web interface, activate live streaming and copy the rtsp url
9. ffmpeg -i rtsp://192.168.10.11:8086 -an -c:v copy -flags global_header -bsf dump_extra -f rtp rtp://192.168.10.1:8080

Here, the ip after rtsp is obtained for camera, the one with rtp is for host device (nano in this case)

10. If all is right, you should see frames being captured. How to open them? I couldn’t get janus streaming.html working but i did manage to capture rtsp frames to python.

Part C: Capturing RTSP frames to openCV:

import os

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "dummy"

import cv2

vcap = cv2.VideoCapture("rtsp://192.168.10.11:8086/out.h264")

while(1):

ret, frame = vcap.read()

cv2.imshow('VIDEO',frame)

cv2.waitkey(1)

Part D: Alternative plugin usage:

1. Install RTSP plugin: https://pluginstore.theta360.com/plugins/com.sciencearts.rtspstreaming/ Instructions on how to install plug in are given there.
2. Python Code:

import os

os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = "dummy"

import cv2

vcap = cv2.VideoCapture("rtsp://192.168.10.11:8554/live?resolution=640X320")

while(1):

    ret, frame = vcap.read()

    cv2.imshow('VIDEO',frame)

    cv2.waitKey(1)