HDR plugin to automatically create EXR file. For VFX use

codetricity · May 24, 2019, 4:44pm

HDR2EXR Plug-in Now available in the THETA Store

Everyone can download and install!

Developer mode not needed.

Description

HDR2EXR is a plug-in for RICOH THETA V to help visual special effects professionally easily grab lighting conditions on-set. Just by pressing the shutter button once, the HDR2EXR plug-in will take 11 bracketed pictures and merge them to one HDR file. The main use is intended for on-set VFX HDR capture.

Details

HDR2EXR measures on-set lighting conditions and bases bracketing on that specific lighting situation. It first takes 1 auto exposed picture to determine basic exposure settings. Based on that information, it sets the lowest ISO and then starts taking pictures.

For each bracket it takes 3 pictures and averages them together to reduce noise. It takes 11 brackets with a 2.5 stop increase. This should be enough to capture almost every lighting situation. Unfortunately, very bright light (like the sun) is still visible with lowest shutter times and ISO. This maybe can be fixed on the new RICOH THETA Z1 with higher aperture settings. This version of the HDR2EXR plug-in does not work with the Z1.

Finally, HDR2EXR automatically merges those 11 pictures into one EXR file ready to be used in NUKE, MAYA, etc. Creation of the EXR file is done by utilizing OpenCV HDR libraries.

Installation

And make sure to set the plugin as the default plugin to use.

How to use?

Start the plug-in by holding down the Mode button for 2 seconds. The little LED will turn white. And the Wi-Fi LED will turn Magenta.
Place the camera in the chosen location (use a tripod, shooting handheld will lead to crappy pictures) and push the shutter button. You have 5 seconds to run away and hide, else you are in the picture.
The Wi-Fi LED will turn green and the THETA will make picture taking sounds. It takes 34 pictures (1 to measure lighting and 3 times 11 brackets).
After the picture taking the Wi-Fi LED will blink red and blue. You can now move or pick up the camera. It is busy merging the pictures. This takes about 1-2 minutes. When it is done it will make a sound and the Wi-Fi LED will turn Magenta again.
Connect the THETA to a computer to download the pictures.

Good to Know

After running the plug-in, reboot your camera. The new files will not show up properly on Windows or Mac without rebooting.
In order to be copied and moved over to Windows, the EXR file is first created with a .JPG extension. It comes in this format: “EXRFILER05071408_removethis.JPG”. On Windows, using File Manager, copy over to your computer and then change the extension to .EXR. On a Mac, use the app Image Capture. Both the EXR file and the “EXRFILER05071408_removethis.JPG” will be available to copy over.
The default sleep setting for THETA is often 3 minutes. You should use the RICOH THETA mobile app to set sleep to OFF.
HDR2EXR tries to keep the ISO as low as possible but also the exposure time. When exposure gets above 1 sec, it increases the ISO until it runs out of ISO and then increases exposure time again.
This version works with OpenCV 3.4.4. I ran into to some problems with 4.0 which I couldn’t fix right away. It also generates a tonemapped jpg, just for fun. I haven’t been able to get this jpg to show up in the THETA iOS app. Don’t know why, maybe someone can help?
If you want to build it for yourself make sure to change the file paths in the Android.mk file (in the app folder).

Credits

The picture taking part is largely based on the work of Ichi Hirota’s dual-fisheye plug-in GitHub - theta360developers/original-dual-fisheye-plugin: Ichi Hirota's dual-fisheye plug-in for THETA V. Sample code for the THETA developer workshop. significant modifications by community
The integration of OpenCV is a combination of RICOH Blog Post: "Running OpenCV in Your RICOH THETA" and Setup OpenCV SDK in Android Studio project and a lot of trail and error!
The HDR part is based on https://www.learnopencv.com/high-dynamic-range-hdr-imaging-using-opencv-cpp-python/
Feel free to change, improve and of course use!
Let me know what you think and run into!

FridayMarch26th · May 29, 2019, 10:28pm

Yo,

The following is a post of unfortunately limited utility, but just to offer a round of applause for building such a useful tool.

I’m a 3D artist drawn to the theta for making speedy HDR maps on set for post production use. With the new Z1 hardware, what’s not to love?.. Well, the accompanying software. The promise (or perhaps I misunderstood) of easy HDR creation at the click of a button is tantalising, but given the current SW experience it’s a promise that isn’t delivered.

Awkward tooling, poor documentation (my favourite thus far has been a reference “See the help”… in the help), and a few errors that I’m currently unable to deal with, and I cannot at the time of writing take my exposure bracketed snaps and transform them into something useful. Hugely, hugely frustrating. Again, great hardware on paper, and the potential for a super-useful solution to HDR creation on a budget, but I just don’t feel as though the current SW experience has my back.

HOWEVER! The existence of this tool fills me with hope that the answer to my current nightmare may one day be realised.

Thanks again,

codetricity · May 29, 2019, 10:32pm

@FridayMarch26th, thank for the note of hope.

Please note that the plug-in does not currently work on the Z1. I think that only a minor modification is needed to the threading to the larger Z1 image sizes. Do you have a THETA V or a Z1?

If you have a THETA V and use this plug-in, make sure you understand that you need to change the extension from *.JPG to *.EXR on your local computer.

If you have any questions about the plug-in, please post them here.

Let’s keep the hope alive!

FridayMarch26th · May 29, 2019, 10:47pm

I’m a Z1 owner (…of around two hours), and yes, I note that the Z1 isn’t yet supported by this tool.

Really, I just wanted to note somewhere that the current Z1 software is not good, while sounding up beat about the future of the product.

So fingers crossed for that future. Let’s hope that as time passes development continues and the SW catches up with the HW.

codetricity · May 29, 2019, 10:57pm

@FridayMarch26th, thanks for the note. It did sound upbeat and hopeful.

I can provide a little assistance with the following:

if you want to provide more specifics on the limitations of the Z1 software, I can pass the information to the product manager. In particular, are you referring to the workflow of using the mobile app to take bracketed shots (how many?) and then using Photoshop / Lightroom (or maybe specialized software) to create the HDR file. From there, do you put it into something like MAYA?
as Kasper was nice enough to open source the software, I was able to test it on the Z1. @jcasman and I may be able to help Kasper figure out where the problem is. I think the problem is minor. If Kasper fixes the problem, I can definitely help him to load an updated plug-in to the store

Additional note. I only looked at the problem for 10 minutes, but I think it may be related to the larger Z1 images taking up more processing cycles and I think it may just require the “merging” process to be run on a background thread (which is fairly simple to implement). I’m hopeful that it’s an easy fix. :-0

jcasman · May 29, 2019, 10:58pm

@FridayMarch26th, thanks for the feedback and all the details. Can you point out this self-referencing help link that you’re referring to? Is this in the RICOH THETA mobile app? The HDR2EXR documentation? Where did it come up?

rios · May 31, 2019, 3:32pm

Will it work with version 3.00.1 firmware?
I was able to shoot, but removethis.JPG is not generated.

codetricity · May 31, 2019, 3:42pm

The reverse of this technique could work for the EXR file.

Here’s a skeleton to run an HTTP server inside of the camera. It would need to be adapted for the specific file format of EXR.

Here’s an article and longer example of using a HTTP server that runs inside the camera.

codetricity · May 31, 2019, 4:06pm

There may be a problem with firmware 3.00.1. I’m checking it now. The plug-in is stopping with a pi-pi-pi error sound, which is not normal behavior. Asking @jcasman to help with testing now.

I will also test with firmware 2.50.1, which was the last version of firmware tested before submission.

test with firmware 2.50.1

Test 1

result: successful
firmware: 2.50.1
reboot: yes

Test 2 - no reboot

result: stuck on red flashing LED with no blue LED mixed in. No HDR file produced
firmware: 2.50.1
reboot: no

took out of plug-in mode
set back into plug-in mode
took three attempts to put into plug-in mode (came out of plug-in mode after a second on two attempts)

Test 3

result: success
reboot: yes
firmware: 2.50.1

WiFi LED solid magenta

0:50 flashing green and taking pictures
0:50 to flashing red with blue LED blinking occasionally
2:00 blue flashing
2:30 red flashing
2:40 ending sound. One continuous sound. Solid Magenta
press lower mode button to take out of plug-in mode

Test 4

test condition: delete all old files from camera. reboot power of camera by holding down power button.
result: successful
firmware: 2.50.1
reboot: yes

Wi-Fi LED is solid magenta at start of test

0- 0:50 - flashing green
0:51- 2:05 flashing red with some blue
2:06 - 2:40 flashing blue
2:41 - 3:00 flashing red
3:01 - end sound, solid magenta

Take out of plug-in mode by pressing lower mode button.
Connect to Windows 10 computer with USB cable.

Testing HDR file with Luminance HDR. File appears to contain all relevant light information.

3.00.1 firmware tests

Note: first three tests failed. Plug-in completes all shots per the procedure above. Does not save HDR file to camera storage. Will stop working with “pi-pi-pi” sound

Error using debugger

2019-05-31 04:26:14.488 2454-2454/com.kasper.hdr2exr I/MainActivity: Done with picture taking, let's start with the HDR merge.
2019-05-31 04:26:14.488 2454-2454/com.kasper.hdr2exr D/MainActivity: times length is: 11
2019-05-31 04:26:14.489 2454-2454/com.kasper.hdr2exr I/MainActivity: Merging average pics for denoise.
2019-05-31 04:26:14.614 2454-2454/? D/AndroidRuntime: Shutting down VM
2019-05-31 04:26:14.629 2454-2461/? I/art: Debugger is no longer active
2019-05-31 04:26:14.629 2454-2461/? I/art: Starting a blocking GC Instrumentation
2019-05-31 04:26:15.175 2454-5696/? W/art: Thread attaching while runtime is shutting down: Binder:2454_3
2019-05-31 04:26:15.175 2454-5696/? I/AndroidRuntime: NOTE: attach of thread 'Binder:2454_3' failed

jcasman · May 31, 2019, 9:20pm

I’ve tested the HDR2EXR plug-in on a THETA V running firmware 2.50.1. This means it’s currently one update behind the 3.00.1 firmware.

The plug-in works well for me in multiple tests over several hours.

Downloaded and installed HDR2EXR from the store using USB
Set it as the boot plug-in using the RICOH THETA desktop app using USB
Set THETA to Plug-in Mode - Main front LED goes from blue to white and the Wi-Fi LED turns magenta (pink)
Shutter button starts the plug-in, it takes 34 images - This takes a little time - Approx 1 minute
When it’s done taking the pictures, while is it busy creating the HDR file, the Wi-Fi LED blinks both blue and red - This also takes a little time - Approx 2 minutes
When the HDR file is created, the Wi-Fi LED returns to magenta (pink) - I did this multiple times in a row - Sometimes I waited several minutes or sometimes I pressed the shutter button again almost immediately - Works fine
Note: You need to reboot the THETA to see your HDR file - This is true for both Mac (using Image Capture or RICOH THETA File Transfer for Mac) and Windows (using File Manger)

Summary: HDR2EXR works for me using 2.50.1 firmware. If you want to use it, I believe it is not “easy to use,” but it does, with just one button press, nicely find the lighting settings and then use those to set the ISO for 11 bracketed images, and then produces an HDR file.

Getting that file off the THETA requires a reboot. And you need to understand that you then pull the file off the camera, rename the extension (format is this: “R05311154.EXR_removethis.JPG”) from .JPG to .HDR in order to get what you want. It’s not a big challenge, but it requires following the steps.

Example from Windows File Manager showing 8 HDR files (“R05311154.EXR_removethis.JPG”) inside the THETA ready to be moved to my computer

Zsolt_Horvath · June 3, 2019, 6:27pm

How can I delete on Mac folders created by HDR2EXR plugin ? ( the files I can delete with Photo App on Mac, but the folders are still on the Theta V.

codetricity · June 3, 2019, 6:31pm

Use the “Image Capture” app on the Mac. Plug the THETA into the Mac with a USB cable. Please post additional questions if you have any problems.

jcasman · June 3, 2019, 7:05pm

@Zsolt_Horvath

Here’s my current understanding:

I cannot delete the folders on a Mac. Using Image Capture (or, as you suggest, the Photos app) I can delete all the pictures and .txt files. The folders will remain. Each folder takes up 4K.

I can see the folders by using the RICOH THETA File Transfer for Mac app, but I am not be able to delete them.

On a Windows machine, I can use the normal File Manager to see the folders and delete them.

codetricity · June 3, 2019, 7:53pm

I’m not using a Mac. What do you mean, “I cannot delete the folders on a Mac?”

Are you suggesting that the file structure of the folder name can’t be deleted? Isn’t there some want to delete it from a Mac?

Zsolt_Horvath · June 3, 2019, 8:13pm

HDR2EXR plugin create folders and put photos into the folders. after downloading the photos I cannot delete the folders.

codetricity · June 3, 2019, 8:34pm

@Zsolt_Horvath, thanks for this clear explanation of the problem. As I don’t have a Mac, I was not aware of this problem. Also, during development, the developers can delete the folders from the camera. It’s good to identify this problem so we can look for a fix.

@jcasman is there no way to delete the folders?

The plug-in can be modified to look for empty folders and delete them, but it would be nice to find a way to delete the folders right now. Otherwise, we’ll need to resubmit the plug-in to the store.

gcorner · June 8, 2019, 10:14am

I am in the same situation as you and keeping my fingers crossed that this tool will run on Z1 as fast as possible.
to all: If i can help in some way please give me some message.

AlexandreDizeux · June 8, 2019, 4:13pm

same for me… I am trying to come back to the previous firmware version, is there a simple way to do that ? I imagine that I have to go through developer mode that I tried to avoid in the first place

Kasper · June 8, 2019, 8:33pm

Hi all,

I’ve pushed a new version to the github. It worked on my V with firmware 3.00. Can you guys test if it works for you aswell?
Indeed it was the shutter_speed_tag in the exif info. Thanx for the heads up!
Also I’ve enabled the z1 tag so maybe that will make it run on the z1. I still haven’t got my hands on one so I can’t test it. If it works it will running in the lower V resolution so it will not add much extra except a better lens.

Hope to test it out on a z1 soon but they are in all sold out pretty soon here in Holland. We’ve got an order standing so hopefullt it will come in soon.

Also I’ve merged it with the plugin store version so all that info should be there!!

Let me know if it works!

Greetings Kasper

jcasman · June 9, 2019, 11:36pm

@Kasper, @AlexandreDizeux, @gcorner, @codetricity,

I tested the new apk that @Kasper uploaded yesterday. It appears to work on both 2.50.1 and 3.00.1.

I am able to test on firmware 2.50.1, 3.00.1 and on a Z1.

I’m using Windows.

2.50.1 - Works
3.00.1 - Works!!!

Notes - not new, all documented, but worth re-iterating

If you’re loading an apk (as opposed to downloaded a finished plug-in from the Store) be sure to set permissions using Vysor or some similar program
Be sure to reboot after running the plug-in to see the files taken by the plug-in

Z1 - Tried multiple times, was not able to install

Installed and setup like this:

But when I click OK, I get this error message: