It is an open source SLAM (simultaneous localization and mapping) framework that advertises support for equirectangular videos like those recorded (or streamed) from a Theta. It supports monocular camera setups, where you use a single camera and stucture-from-motion to estimate the actual 6 dimensional trajectory of the camera.
To be clear, this is a REALLY cool. Projects like this make autonomous robotics MUCH more accessible. The computer vision technology they are using, ORB features for key framing, is a pretty cutting edge approach that has only really become popular in the past 4 years. Technology like this allows a robot to navigate and map its environment autonomously in real time. A single camera is an order of magnitude cheaper than the LIDAR / LIDAR+IMU / camera+IMU setups that are usually used for these kinds of things. I even found a report from someone who claims to have been able to succesfully use the software with a Theta V.
Anybody have experience with this? I am finally settled and think I might pick up a Theta to build a bot and test this out.
Sickkkk. Did you test it with a pre-recorded video, or did you use some method like UVC to stream video to the system live? It seems to me like that would require a bit more work, but to me thats a much more interesting use-case.
At the moment, the UVC webcam technique can’t be used with the THETA on Linux as we don’t have a streaming driver at the moment. I’ve heard that some people on the community on working on a driver.
I’m currently trying to get it working with the z1 but im having trouble. I got the z1 to work with gst_viewer and gst_loopback to get the camera under /dev/video2. The problem I’m having is getting OpenVSLAM to see the video feed. When I run $ ./run_camera_slam , OpenVslam only reads one frame form the video feed and then won’t read anymore. So basically it just freezes after one frame.
ok, so I’ve narrowed down the problem to the fps with v4l2loopback. Seems like loopback is slowing it down because I don’t have something set right. On gst_viewer I get realtime. Before I plug in the camera I use the android app and set the video resolution to "FullHD(1920x960), I’m not sure if this is the right way to change the live feed resolution. Anyways, I plug the camera in and start my laptop and open a terminal in the v4l2loopback dir and enter the command
sudo modprobe v4l2loopback exlusive_caps=1 max buffer=8 width 1920 height=960
then I check my /dev/video* with the command
ls -ltrh /dev/video*
and see that I created crw-rw----+ 1 root video 81, 2 Dec 9 06:32 /dev/video2
I check gts_viewer and see that it works and it does.
Then I enter the ./gst_loopback command from the /gst dir.
I open VLC and select my capture device as /dev/video2 and press play.
VLC displays one new frame immediately, then the next frame about 20 seconds later, and then it seems to get slower but I’m not counting. That’s my problem.
I’m not sure what type of information you would need to help me with this problem so just tell me what you would need to know to help me.
I’m away from home so I can’t test it with qos=false and at 2k resolution right now but I will test it in about 12 hours and report back.
Thank you for your help this is my first linux project ever so I’m very new.
The output of v4l2-ctl in your post is the built-in webcam on your laptop. To see the THETA, most likely on /dev/video1, do this after you start v4l2loopback.
I am hopeful that you will see better framerates when you set qos=false in the C code and then recompile the code.
Feel free to post again. You are several steps away. If you can get decent framerates with v4l2loopback on /dev/video1 (I’m assuming your laptop webcam in on /dev/video0), then test it with vlc or gst-launch-1.0 or another video player before moving to SLAM.
You can get the IP addresses of the devices on your network with:
sudo arp-scan --interface=eth0 --localnet
Change the interface to your Wi-Fi card. The example above is Ethernet.
On a Jetson Nano, I am receiving the RTSP stream with this and viewing it as equirectangular. The pipeline is specific to a Jetson. However, you can get a basic idea of the concept.
Hi @ryleymcc and @craig . I’m considering using a ricoh theta V or the same Z1 for the same purpose but I have some doubts. I’m not working with a NVIDIA, I don’t know if that will be a problem to get the stream from the USB. Also, is the latency a problem when creating the map in real time?
If you can tell me your experience using the Theta Z1 to create a map and localize in it, that would be very helpful. Like, for a small map it works fine but for a larger one it is impossible or the way to tune the configuration parameters for the theta to use it on OpenVSLAM properly.
