Using Sony MESH IoT Plug-in with RICOH THETA V

jcasman · October 11, 2018, 12:28am

Using Sony MESH IoT Plug-in

On the RICOH THETA Plug-in Store, there are a bunch of cool Partner Plug-ins that you can download and install on your THETA V and just… use as much as you want.

The Sony MESH IoT Plug-in allows you to connect to two cool little IoT devices and remotely control your THETA. Either as a remote shutter with no wires using the “Button Block” or by have it take pictures or video with a motion sensor making it very easy to set up camera trap for pets or wild animals.

It doesn’t take much at all to set up, and, wow, it feels clean and easy to use. You can do both images and video. The Sony MESH blocks themselves are small and tight and very impressive to use.

Translation of Support Page

In the Plug-in Store there’s a support link, but it seems to point to all sorts of Sony MESH support issues. I couldn’t find a simple explanation of how to set up the plug-in with the MESH blocks.

But I looked around, and I found this, a Japanese version of How to Use the MESH Plug-in for RICOH THETA. Bingo! Now I just needed a translation. As luck would have it, I speak Japanese. So, this is my informal translation. I think it’s solid, cause the original is simple and straightforward, but it’s not official. Posting it here to help out others!

How to Use the MESH Plug-in for RICOH THETA

Overview

You can control recording with a THETA V by using either the MESH Button block (on/off switch) or a Move block (motion sensor). It is possible to control without using the MESH application, so it can be used even without a smartphone or tablet.

How to Install

Please install from this site: MESH Plug-in for RICOH THETA | RICOH THETA Plug-in Store

Please check here for the plug-in explanation: https://pluginstore.theta360.com

How to Use

Press and hold the “shooting mode button” on the THETA V and change THETA V to plug-in mode. When changed to plug-in mode, the THETA V’s camera status LED lights up in white. From here on out, all operations are occuring pairing mode.

How to do Pairing

When using the plug-in for the first time, it automatically searches for nearby MESH blocks. The Wi-Fi LED on the THETA V will be blinking quickly. When the MESH block is paired with THETA V, the Wi-Fi LED stops blinking, and the "shooting mode lamp" of THETA V lights up in either still image or movie mode. (The shooting mode can be changed by pressing the "shooting mode button" on the THETA V.)

Once pairing is done, it will automatically reconnect to the pairing MESH block at plug-in startup.

To change pairing, press and hold the "shutter button.” The plug-in will automatically search for nearby MESH blocks again.

The meanings of the Wi-Fi LED while in plug-in mode are as follows:

On: Connecting
Slow blinking: Attempting to pair with a MESH block
Quick blinking: Trying to pair with a new MESH block

Only one MESH block can be paired at a time. To check whether it is connected on the MESH block side, press the MESH block icon part twice. If the status lamp lights up in blue, it is connected. If it is blinking, it is not connected.

How to Take Pictures

You can take pictures with the following techniques:

Button Block

Still image shooting mode: Press the button to shoot.

Movie shooting mode: Press the button to switch between shooting start / stop.

Move Block

Still image shooting mode: It will shoot when the motion sensor is triggered.

Movie shooting mode: When the motion sensor is triggered, shoot a movie for 15 seconds. When the motion sensor is triggered during shooting, the shooting time is extended by 15 seconds. If there is no further motion, shooting stops.

The reaction interval of the move block is 3 seconds.

Customizing More…?

It is also possible to connect to THETA from the MESH application using MESH SDK. For details, please visit the following site.

Jesse’s Tips for Installing the Sony MESH Plug-in

Use the RICOH THETA desktop app. It is quicker and easier than installing a plug-in through the mobile app.
Make sure you’ve got the latest firmware. This can also be upgraded through the desktop app. The plug-in webpage says Firmware version 2.30.1 or later.
The first time through the Wi-Fi status button blinks fast, but it wasn’t connecting. The key was turning on the Sony MESH block!

Turning on the Sony MESH block

Press and hold the icon button on the bottom half of the MESH block for 2 seconds. The status lamp (between the top and bottom halves) with turn on indicating that the MESH block is on.

Icon%20Button

Join the THETA Partner Program to Start Building Your Own Plug-in

codetricity · October 11, 2018, 6:37pm

I can’t connect my THETA V to the MESH Button Block. Please help.

Plug-in is installed.

Block is fully charged.

See my process below.

Please help.

jcasman · October 11, 2018, 10:22pm

I am able to connect my Sony MESH Button Block to my THETA V. Here’s a video of me doing it. It may require a SECOND on/off button press (bottom green half of the button) to get it going. The plug-in has nice clear connect and disconnect sounds as part of the process. I don’t see this in the documentation, but I’m able to replicate it regularly.

codetricity · October 11, 2018, 11:17pm

Did you use the MESH mobile app at any point?

Why do you have a USB cable plugged into the camera in the video? Is it connected to your computer?

jcasman · October 11, 2018, 11:23pm

No, I did not. The Sony MESH mobile app is not needed for the plug-in.

Actually, when I first couldn’t turn on the block (I didn’t even know that I needed to turn it on, though of course it needs power) I started to go through the documentation for the MESH mobile app just looking for hints. That’s when I ran into the power on info.

This is a screenshot of the information you get when you first try the mobile app:

codetricity · October 11, 2018, 11:37pm

so, you turn it on and the block automatically pairs with the THETA with bluetooth?

It’s absolutely amazing that there is no documentation in English. Maybe contact the people you know at MESH and indicate that most people are having problems as a result of no documentation.

It would help with basic stuff such as turn on the block. it’s not obvious how to do it.

Also, I believe the THETA V needs to have Bluetooth enabled and the block can’t be paired with another device.

Maybe they hope that the setup is so easy that no documentation is needed? However, that’s not a correct assumption.

mcchaus · October 12, 2018, 2:48pm

What kind of range do you get with the Sony MESH block?

jcasman · October 12, 2018, 2:53pm

Here’s the technical specs I have for it. According to Sony MESH, it says “Line of sight distance - 30 feet”

Source is https://support.meshprj.com/hc/en-us/articles/212603007-How-does-MESH-Button-work-

Technical Specifications:
Size (width, height, depth): 24mm, 48mm, 12mm
Weigh: 13g
Communication method: Bluetooth standards Ver. 4.0 (Bluetooth Low Energy)
Maximum communication distance: Line of sight distance - 30 feet

Power: built-in lithium-ion battery
External interface Micro USB (for charging only)
Operating temperature limit: 0C~35C
Battery charging time: approximately 1 hour (it can be influenced by temperature and usage.)
Corresponding devices: Please Check: http://support.meshprj.com/hc/en-us/articles/212601267-Which-devices-can-operate-the-MESH-app-

I will be testing it more personally this weekend both outdoors and indoors. I’ll add any information I get here.

codetricity · October 12, 2018, 8:37pm

I was able to get the MESH Button Block to work. See this video for more information.

I ended up turning Bluetooth on the THETA V on and off. I’m not sure if that was related to it working or not.

The latest lesson learned today from @jcasman is that pressing the shutter button of the THETA V will switch the pairing of the blocks. This means that if the THETA V is connected to the Push Button block, then you need to press and hold the shutter button before it can connect to the motion detector block.

@jcasman got the motion detector block working today.

Pre-shoot Setup

On-Site Setup

Action Shot

jcasman · October 13, 2018, 12:08am

That is one hell of an action shot

codetricity · October 15, 2018, 4:02pm

Did you progress further in your tests?

jcasman · October 16, 2018, 9:07pm

I set up the RICOH THETA running the Sony MESH plug-in to take pictures of birds. I like bird watching, I thought this would be a cool use for the camera. I believe it’s commonly called a “camera trap.”

The results have been limited so far. I need to do more testing. But here’s a couple preliminary comments:

I tried several different configurations, including a hummingbird feeder. Position of camera and Sony MESH motion sensor button was tenuous. But most importantly, no one showed up. I guess hummingbirds don’t try out the new restaurant in the neighborhood right away.
So, I went to my local community garden, and put out bread crumbs. This worked better. The configuration with the camera was simple, I just taped the motion sensor directly onto the camera.

Setup was a chair, and breadcrumbs. (Eventually I put the camera down closer to the bread crumbs on the ground.)

It first caught some things other than birds

But eventually, it worked!

Caveats and lessons

Wind in the bushes or other movement can trigger the motion sensor. Depending on your setup, this can produce a bunch of unwanted pictures. Then again, the 19GB of internal memory can handle 4,800 photos, so maybe this is not an issue.
Sony MESH motion sensor button has some specific specs that you should be aware of. The one that surprised me is that motion is detected from side-to-side. If the bird moves directly towards the sensor, it’s possible it does not trigger it. Details here:

https://support.meshprj.com/hc/en-us/articles/212120438-How-does-MESH-Motion-work-

From the website:

MESH Motion uses infrared heat to detect motion which can impact accuracy when the tag is moved or placed in close proximity to lots of human or animal movement. MESH Motion only detects motion that crosses paths with the tag, so motion such as walking straight towards the tag or away from the tag will not be detected by the sensor.

Definitely need to protect the THETA. This is no different from any normal THETA use.
Definitely need to take into consideration normal photographic settings. In my case, I was taking pictures in the shade which ultimately makes the birds harder to see. You can do some post-processing and make the pictures more clear, but a better initial setup would save you effort later.

Increased%20brightness%20and%20contrast

codetricity · October 16, 2018, 9:22pm

Were the birds large enough to trigger the camera? How many pictures did you take in order to get that picture of the birds?

I would think that the sensor would benefit from larger animals like caribou.

Did you try the 4K video setting instead of still image?

jcasman · November 30, 2018, 8:41pm

Did a little more testing over the Thanksgiving holiday. Installed the THETA on top of a bird feeder. Was just testing placement first, without setting up the motion sensor. And, actually, that’s as far as I got. I concluded that placement is everything. Will need to work on a more central spot in order to have satisfying results.

The Rig

In Situ

2D Picture Showing Potential

Example Image

Looks better in a 360 viewer, like the RICOH THETA desktop app, where you can move around and zoom in easily.

Details

Conclusion

The birds on the feeder (yellow top) ended up being lower down mostly and somewhat obscured. Intend to install tripod with THETA directly between the two main feeders for optimal photo-taking.

codetricity · November 30, 2018, 9:16pm

Click on full-screen to zoom on the birds

jcasman · November 30, 2018, 9:41pm

Thank you, that’s much better!

codetricity · November 30, 2018, 9:48pm

Have you considered using the Tensorflow Object Detection to trigger the camera to take a picture of the bird?

It may help you to avoid random pictures of falling leaves or people walking past the motion sensor.

https://bitfusion.io/2016/08/31/training-a-bird-classifier-with-tensorflow-and-tflearn/

codetricity · December 4, 2018, 5:56pm

I captured a bird in flight with my long video plug-in.

At 30 fps, I’m able to analyze the flight path and wing movements.

You can see the bird at 0:22 of this video.

jcasman · December 4, 2018, 5:58pm

I’m pretty sure that’s a Dark Eyed Junko. Does it act like a flycatcher? Then it definitely is.

codetricity · December 4, 2018, 6:01pm

I think it came out of the sky to attack the camera, but then veered off when it realized it wasn’t anything interesting. You can see the behavior yourself in the video clip. Let me know what you think it is.