Introduction
Original article appeared in Japanese here.
This is Yuuki_S from Ricoh. RICOH makes a camera called RICOH THETA that can shoot 360 degree images and videos. The RICOH THETA V and Z1 use Android as the OS, so you can customize your THETA as if you were making an Android application. The customization function is called a “plug-in.” (See the end of this article for details).
The Ghost Town effect, a unique photography technique similar to long exposure, is interesting.
I wanted to take a picture like this myself, so I built a THETA plug-in to do it.
What do you think? The people disappear and it appears to be a ghost town.
I will show you how to make this THETA plug-in. If you want to just jump to the final image instead of how to make it, please see the examples at the end.
Introduction to OpenCV
If you want to perform image processing efficiently, using OpenCV is the quickest route. OpenCV can be used with THETA plug-ins, and past articles have covered how to install it:
Running OpenCV in Your RICOH THETA. This article introduces how to handle OpenCV in NDK written in C ++. In this article, I will introduce another method, statically linking the library and handling OpenCV in Java.
It was roughly divided into three steps, and it took about 5 minutes to introduce. It’s simple.
- Getting OpenCV Android Pack
- Importing OpenCV into your project
- Editing Source Code (Implementing OpenCV Interface)
1. Getting OpenCV Android Pack
Download the latest version of OpenCV version 3 “Android pack” from the OpenCV Releases page.
Note that version 4 does not match the required Android API level with THETA, so there may be many features that do not work. This time, I used 3.4.12.
Extract the downloaded Zip file and place the folder in any location. I’ll link to this folder in Android Studio later, so I think it’s a good place to find it. I created a folder called OpenCV directly under the C drive and put it in it.
2. Importing OpenCV into your project
After opening the project in Android Studio where you want to import OpenCV, select “File”-> “New”-> “Import Module” from the menu, and select “C: / (where OpenCV is located) / sdk / java” in “Source directory”. Hit enter.
After adding it, the integrity of the set of project files is checked. You will be notified that there is a problem with the following line in AndroidManifest.xml for “openCV3412”, so delete this line. (It is line 7. In the figure below it has already been deleted.)
In addition, please modify “compileSdkVersion” and “targetSdkVersion” of build.gradle (Module: openCVLibrary3412) to “25”.
android {
compileSdkVersion 25
～shortened～
defaultConfig {
～shortened～
targetSdkVersion 25
}
～shortened～
}
Some Android Camera2 APIs used by some OpenCV version 3 functions are not supported by THETA. Here’s how to ignore the inconsistencies and let the build go through. I did this because it does not affect the basic image processing functions and is simpler than modifying the library side to resolve inconsistencies.
Next, create a folder called “jniLibs” in the specified location in the project, copy the file under “arm64-v8a” (the library with the extension so) from the OpenCV library, and save it there.
Copy source: C: / (where OpenCV is placed) / sdk / native / libs / arm64-v8a
Copy to: C: / (where the project files are) / app / src / main / jniLibs / arm64-v8a
Next, configure the settings to enable this file. From the Android Studio menu, open “File”-> “Project Structure” and from the left, select the “Dependencies” tab, then the “app” tab, and then click the “+” under the word “Declared Dependencies”. Choices will be displayed, click “Module Dependency”.
Check “openCV Library 3412” in the displayed dialog and press OK. This completes the importing OpenCV into your project.
3. Editing Source Code (Implementing OpenCV Interface)
Finally, I will describe loading OpenCV in the THETA plug-in. Add the following to onResume so that it will be loaded when the plug-in starts.
protected void onResume() {
super.onResume();
if (!OpenCVLoader.initDebug()) {
Log.d(TAG, "Internal OpenCV library not found.");
} else {
Log.d(TAG, "OpenCV library found inside package.");
}
～shortened～
}
You have now completed handling OpenCV from the Java side.
Implementing Ghost Town Effect
This time, I wanted to check the result in real time when shooting outside, so I based it on the project from the following article where images can be previewed on a smartphone.
Make live previews easier to handle with THETA plug-ins (in Japanese).
The method is to apply processing to the image acquired by live preview. Although it is easy to check on the spot and easy to implement, it has the disadvantage that the resolution is about full HD (1920x960) because the image is processed using live preview. I decided on this method to prioritize convenience, but I feel that it is actually more normal to take time-lapse photography and process it.
However, I think that the image processing of the effect will be the same even if you use time lapse.
The implementation can actually be done with one OpenCV function.
This function can obtain the moving average of continuous images and is used to extract the background image by detecting moving objects based on background subtraction. (Initially, I completely forgot about the existence of this function and tried to implement it from scratch. I’m glad I noticed!)
The code that uses this is as follows. The actual effect processing is only the one line mentioned above, and before and after that is only the conversion of the image format.
public void drawGTEffectThread() {;
new Thread(new Runnable() {
@Override
public void run() {
android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND);
Mat averageImage = new Mat(960, 1920, CvType.CV_32FC(4));//background image
Bitmap back_frame = Bitmap.createBitmap(1920,960, Bitmap.Config.ARGB_8888);
while (mFinished == false)//live preview state
{
byte[] jpegFrame = latestLvFrame;
if ( jpegFrame != null )
{
if(toggle_GTEffect == true)
{
Bitmap frame = BitmapFactory.decodeByteArray(jpegFrame, 0, jpegFrame.length);//Bitmap conversion of preview image
matFrame = new Mat(back_frame.getHeight(), back_frame.getWidth(), CvType.CV_8UC3);
matFrameBG = new Mat(back_frame.getHeight(), back_frame.getWidth(), CvType.CV_8UC3);
Utils.bitmapToMat(frame,matFrame);//Mat conversion of preview image
Mat matFrame_f = new Mat(960, 1920, CvType.CV_32FC(3));//Prepare float Mat
matFrame.convertTo(matFrame_f,CV_32FC4, 1/255.0);//Convert preview image to float
Imgproc.accumulateWeighted(matFrame_f,averageImage,0.03);
averageImage.convertTo(matFrameBG,CvType.CV_8UC3,255);//Convert background image from float to Mat
Utils.matToBitmap(matFrameBG,back_frame);//Convert background image from Mat to Bitmap
ByteArrayOutputStream baos = new ByteArrayOutputStream();
back_frame.compress(Bitmap.CompressFormat.JPEG, 100, baos);//Jpeg conversion for preview display
latestFrame_Result = baos.toByteArray();
Map<TextArea, String> output = new HashMap<>();//Display status on Z1 OLED
output.put(TextArea.MIDDLE, "");
output.put(TextArea.BOTTOM, "Processing...");
notificationOledTextShow(output);
}else{
Map<TextArea, String> output = new HashMap<>() ;
output.put(TextArea.MIDDLE, "");
output.put(TextArea.BOTTOM, "Stopping...");
notificationOledTextShow(output);
latestFrame_Result = latestLvFrame;
}
}else{
try {
Thread.sleep(125);
} catch (InterruptedException e) {e.printStackTrace();}
}}}}).start();
}
After that, call this thread with onResume and switch between false and true for toggle_GTEffect from the button, and you’re done.
One thing to note is that this implementation has a fixed input resolution for the live preview, so an error will occur if different resolutions are entered. Therefore, you need to specify the preview format as follows.
protected void onResume() {
～shortened～
//Start LivePreview
previewFormatNo = GetLiveViewTask.FORMAT_NO_1920_8FPS;
～shortened～
}
In addition, change the JavaScript on the web display side. In the initial state, 2 (960x480) is selected.
var PREVIEW_FORMAT = 5;
With this, you’re all set!
Checking Operation & Examples
Set up your THETA outdoors and connect to your smartphone.
If you access 192.168.1.1:8888 from your smartphone browser, the preview will be displayed as shown below.
After that, if you turn on the effect, you can get a photo with only the background. I really like how it emerges slowly.
Save the image from the browser. Just press and hold (on iPhone) and select “Add to Photos.”
Here is an example taken in a place with a little more people.
People in motion are nicely hidden. However, due to this technique, people who have stopped for a long time show up.
Since it does not look great in equirectangular, I edited it with THETA + app.
Conclusion
I was able to make a THETA plug-in that can easily apply the Ghost Town Effect. It’s fun to manipulate effects (image processing) that you come up with and see the results on the spot.
Since OpenCV can be used quite easily, please try to make a THETA plug-in that uses your own favorite effects.
About the RICOH THETA Plug-in Partner Program
If you are interested in THETA plug-in development, please register for the partner program!
Please be aware that the THETA with its serial number registered with the program will no longer be eligible for standard end-user support.
For detailed information regarding the partner program please see here.
The registration form is here.