The THETA S is widely used to take 360 videos from a drone. The most common usage is to store the videos on the THETA and then transfer the videos to a computer for processing after the drone lands. The 360 video is then uploaded to a video distribution site like YouTube where people can view the 360 video in a headset or on their computer.
It’s cool. It works. It’s easy. Highly suggest you simply mount a THETA to your drone with a 1/4" by 20 pitch bolt and save the video to the camera.
If you stumbled upon this article and just want to save the video file to the camera, you don’t need to read this article.
If you’re at this site because you’re a hacker and want to push the envelope, you’re a pioneer. Read on.
We often get questions about live streaming a 360 video from a drone to a person on the ground in real-time. The short answer is that as of August 2016, I have never heard of it actually working. It could work and it’s very possible that someone has it working. If you find a working implementation, let me know.
The biggest problem with using the THETA S for streaming from a drone is that the THETA S (as of August) cannot transmit a 720 video stream over WiFi or a radio transmitter.
It only streams over a USB or HDMI cable. This means that you’ll either need to put a small computer on your drone to accept the video stream using USB and then transmit the stream over WiFi to the ground or convert the HDMI signal into an analog signal for input into a radio transmitter.
At this point, you may think, “This is not what I expected, I’ll wait for the streaming technology to evolve a bit. I’ll just store the files on the THETA.” This is fine. You may even want to use two cameras on your drone, a lightweight one for navigation and a THETA to record the video.
Once again, I recommend you save your video to the camera and don’t build your app to depend on live-streaming. You can get good results.
For hackers, the most exciting implementation would be a flying drone streaming at 720p or higher. That’s the most problematic. Consider a land-based car first.
It’s theoretically possible to stream 360 video using the THETA from a drone and I’m sure it’ll happen in the near future.
Here’s the problems with a flying drone:
- weight of the computer connected to the THETA on the drone
- weight of the power source for the computer (if you want to stitch on the drone)
- lack of easy to use viewing software for mobile phone apps (though development is happening incredibly fast)
- lack of easy to use stitching software to convert from dual fisheye to equirectangular
With all these challenges, you may want to just save the videos on the drone.
In live-streaming mode, the THETA S will output a dual-fisheye video with HDMI or USB.
This is dual-fisheye:
Transmission Using Radio Transmitter With HDMI Output from the THETA
Most quadcopter first person video (FPV) feeds use a radio transmitter to send either PAL or NTSC analog video formats. The radio transmitter has a range of several kilometers and would send the dual-fisheye video feed in MPEG-4 AVC/H.264 format to a receiver.
This video gives an overview of using a cheap security camera to transmit video using radio transmitters.
The video signal into the transmitter is analog. Remember the round yellow cables that used to connect to video game consoles before HDMI became popular? Those were analog cables.
The HDMI video signal from the THETA is 1920x1080 at 30fps uncompressed YCbCr format. How do I convert this to analog?
Drone hobbyists can theoretically use an Aerial FPV HDMI to AV Analog Signal Converter.
Here’s another example.
If this works, the video transmitted will be in dual-fisheye mode. You’ll still need to take the video from the receiver and convert it to equirectangular.
This is a bit problematic as the video you get out of the receiver will be in dual-fisheye.
You can then theoretically take the output of the video receiver and convert it into USB to get it into your computer. Seems like a real challenge unless you have the parts lying around and have that urge to hack.
Live Streaming From a Drone Over WiFi with USB Dual Fisheye Output From the THETA
Using the USB output, the THETA can live stream a MotionJPEG format video at 1280x720 at 15fps.
The robot will need to have a small computer like a Raspberrry Pi zero or RPi3 to accept the video stream from the THETA with a micro USB cable. Once the stream is on the small computer, you can then broadcast it from the small computer in real-time over a network connection like WiFi. As the range of the WiFi from the Raspberry is limited, this may not be feasible for many drone applications. This strategy may be a dead end. The main advantage of this strategy is that we know a Raspberry Pi can transmit a video signal to another device and we have working applications to transmit and view the video. The distance problem may be a show stopper.
For example, a Raspberry Pi Zero connected to the THETA with USB that then transmits the video stream using WiFi to a mobile phone in a VR headset on the ground.
In this scenario, the mobile phone would need to then take the dual-fisheye video stream, convert it to equirectangular and then add navigation.
RICOH does not have sample code to do this precisely. The closest approximation is this set of sample code for video streaming and viewing with JavaScript programs. Full source code is available.
The sample code uses JavaScript to convert the dual-fisheye video stream into equirectangular and then adds navigation. There are three programs as part of the sample application.
The one that drone hackers should first look at is oneway-broadcast. The reference platform is a Raspberry Pi running Raspbian Jessie with Node.js. It’s likely that you can set up the stream from the drone and use a Raspberry Pi with WiFi to transmit the video stream to a receiver for watching.
It’s very tempting to try and get something usable to work, but I also think it would take time. It’s an exciting time to be on the cusp of all the developments in 360 video and see the progress each month.
Other mounting techniques.
Again, simply mounting the THETA onto your drone will yield interesting footage. You can use Adobe Premier Pro or another video editor to change the audio. Adobe Premiere Pro started supporting VR workflows in June of 2016.
For other projects involving live streaming with the THETA, take a peek at using THETA live view with Unity, this demo about WiFi streaming with THETA and Unity, and this tutorial on THETA live streaming on YouTube.
Additional Information
DroneKit and Onboard Raspberry Pi
My colleague @tyler.edell recommended that I look at DroneKit. There’s a wealth of information on collecting and sharing drone information. Under the section, Companion Computers, the Raspberry Pi is listed as a supported computer with information on connnecting the RPi with MAVLink.
This section on using the Raspberry Pi as a bridge looks interesting. If you have a Raspberry Pi onboard your drone for other things, maybe it’s possible to use it for streaming as well?
GPS Information
gpsInfo can be attached to your THETA images with the API. The THETA does not have a GPS in it, but you can attach GPS information to your pictures.
{
"gpsInfo": {
"lat": 23.532,
"lng": 23.532,
"_altitude": 999.00,
"_dateTimeZone": "2014:05:18 01:04:29+08:00",
"_datum": "WGS84"
}
}
Mounting Hardware for the THETA
I purchased this hardware from Orchard Supply and Hardware (OSH) in Mountain View, California.
1 pc 1/4” by 20 pitch rod coupling nut
1 pc 1/4” by 20 pitch threaded rod, 3 inches long
2 pcs 1/4” by 20 pitch hex jam nut
It allows the USB cable to be plugged in for power or control while the THETA is mounted.
Alternatives to Live Streaming Dual-Fisheye
This article focuses on streaming a dual-fisheye video from the drone with an onboard Raspberry Pi connected to the THETA with a USB cable and then using WiFi on the Raspberry Pi to stream the video to a mobile device on the ground. The example uses JavaScript on the mobile device to convert the dual-fisheye video into an equirectangular view with headmount navigation (headmount gyroscope navigation is not implemented as of Aug 2016).
There are other strategies
Convert Video to Equirectangular on Your Drone and Broadcast to YouTube
You can use RICOH UVC blender to convert the video from dual-fisheye to equirectangular from the Windows or Mac device that is connected to the THETA with USB. I think this would be too heavy for a flying robot. The advantage is the a Windows or Mac could stream live to a service like YouTube so a million people can follow your drone’s odyssey through jungles, buildings, mountains, and volcanos. The other advantage is that the software for this comes from RICOH and is in wide use. It works. See the article THETA Live Streaming on YouTube for more information.
There is a latency of a few seconds when streaming to YouTube. This streaming is not suitable to control your drone or for augmented reality applications that require real-time human interaction. It is intended for a wide audience to view an event in 360 in close to real-time. Think of watching a sporting event like a drone road race from the perspective of the drone. The small computer on your drone will need to be an Intel multi-core CPU. The real-time stitching is CPU intensive. The RICOH software only works on Mac and Windows. It will not work on Linux.
Onboard Raspberry Pi controls Camera Video and Images (Don’t stream)
Another option is to send a Raspberry Pi up with the drone and have it take pictures or videos based on different inputs or time triggers. You can either use WiFi between the RPi and the THETA to control the THETA or use the USB cable to provide both power and control of the THETA.
In addition to live streaming from a Raspberry Pi and THETA, we definitely know that the THETA API can be accessed over USB from a Raspberry Pi. The picture below is a Raspberry Pi 2 with touchscreen to control the THETA using a USB cable. The THETA can be mounted with standard 1/4" diameter by 20 thread bolts.
Store Long Videos Directly On Your Camera (and void your warranty)
People from the community have voided their warranty and inserted 32Gb Micro SD (Samsung Orange) cards into the THETA.
Seems like a lot of pieces already exist and that an active community of people are hacking the THETA and using it with drones. Live streaming from the drone may be too hard of a hack for most people right now, but there are a lot of things to experiment with, including using an onboard Raspberry Pi to start and stop videos or take pictures based on time or input sensors. The field is so new. Every step you take forward will be new ground that you’ll discover.
New To 360 Videos?
If you’re coming from the drone community and are new to 360 videos, here’s some common questions.
Q: How do people see my 360 videos?
A: YouTube, Facebook, theta360.com and many other sites will host videos for free. They can view it in a web browser or in a headset.
Q: How do people view it in a headset?
A: There are apps, such as the Facebook App and the YouTube App that respond to head motions. Install the app on a mobile phone and put your phone into Google Cardboard. There’s many headsets and adapters to view the videos with a headset. For better video quality, you can also store the video on the phone or headset and view it through a mobile app directly from the local file.
Q: Can people view the 360 video from my drone in a headset in real-time while it’s flying?
A: No. This article discusses a possible hack to get it to work, but no one has a working drone with live streaming as of August 2016
Q: The video file looks like two spheres when I play it
A: Using your desktop Mac or Windows computer, use the THETA desktop application from RICOH. Drag the file onto the application. The application will save another video file as equirectangular. It will look like a distorted rectangular image if you play it in a normal video player. It will look normal and have 360 navigation in the THETA viewer or on YouTube, Facebook, theta360.com
Q: How do I edit a 360 video file?
A: It’s a standard MP4 video. Just use Adobe Premiere or whatever video editing software you normally use. You may need to inject the metadata back in after editing.
Q: Does the THETA have auto-stabilization
A: No.
Q: Can I control the THETA with my smartphone while my drone is flying?
A: It will work if it’s close. It’s likely that the drone will fly out of WiFi range of the THETA. The THETA functions as the hotspot for your phone. Your phone will need to be in range of the WiFi transmitter of the THETA.
Q: Can my drone trigger the THETA to take pictures based on GPS information or object recognition?
A: The API to control the camera is open. You can use a Raspberry Pi to take pictures based on different inputs. The Raspberry Pi can connect to the THETA with WiFi (easier) or with USB.
