Z1 issue: Service Unavailable & Alert

Pixel · February 10, 2021, 6:07pm

I regularly encounter the same problem with the Theta Z1 …

Description of the problem:
After a few series of photos, the device suddenly goes into an unstable state. After a thorough analysis, I found that very often the problem comes after the following sequence:

1- execution of the command: “camera.setOptions” {some options…}, response → {"name":"camera.setOptions","state":"done"}
2- execution of the command: “camera.takePicture”, response → {"id":"813","name":"camera.takePicture","progress":{"completion":0.00},"state":"inProgress"}
3- pause (4s)
4- check the status of the last command {"id":"813"}: response → {"id":"813","name":"camera.takePicture","progress":{"completion":0.00},"state":"inProgress"}
5- pause (4s)
6- check the status of the last command {"id":"813"}: response → {"id":"813","name":"camera.takePicture","progress":{"completion":0.00},"state":"inProgress"}
7- pause (4s)
8- check the status of the last command {"id":"813"}: response → {"id":"813","name":"camera.takePicture","progress":{"completion":0.00},"state":"inProgress"}
9- pause (4s)
→ the sequence continues in a loop… the status is always inProgress ;-(

Once out of the loop (in my case I limit the number of calls to the status service, eg < 10) the next OSC calls will result in either:

a error message → {"code":"serviceUnavailable","message":"Service Unavailable."}...
a device crash with the appearance of a triangle with the text “alert”

The only way to be able to use the device again is to restart it by pressing the buttons “on” & “wifi” on the same time during ~= 11s.

Some important additional notes:

The sequence I have described works approximately 7 times out of 10
The same code works perfectly with the Theta V
The same code worked perfectly a few months ago with the Theta Z1
I performed tests on 7 different devices
I noticed the problem with the following firmware (Z1): 1.60.1 and 1.50.1

Your help is welcome!

craig · February 10, 2021, 6:27pm

Please post your camera options so that we can try and replicate this. @jcasman can report this back to RICOH, but it would help if we can replicate the problem ourselves first.

In particular, I want to use the same DNG(RAW) or JPG resolution as you. There may be other settings you are setting.

Assume that your check confirms that the status is “done”?

You’re doing something like this, right?

Future<String> isDone(String id) async {
  var url = 'http://192.168.1.1/osc/commands/status';
  var data = {'id': id};

  var payload = jsonEncode(data);

  var response = await http.post(url,
      headers: {'Content-Type': 'application/json;charset=utf-8'},
      body: payload);

  Map<String, dynamic> status = jsonDecode(response.body);
  String state = status['state'];

  return state;
}

// at some point in the loop, the return value of the state property in the /osc/status changes to done

    var currentStatus = await isDone(id);

    if (currentStatus == 'done') {
      keepGoing = false;
    }

At some point, the camera stops returning done?

in osc/state, is there an error?

If you can insert state into the loop after four seconds, we may see something.

In your post, the error message below is from Android Studio, right? Not from the Z1, right?

a error message → {"code":"serviceUnavailable","message":"Service Unavailable."}...

How many pictures are in your sequence?

Are you testing it with just 10 pictures? Or, are you taking 100 pictures, one after the other?

Is the camera hot when you touch it?

craig · February 10, 2021, 7:49pm

I just took 100 pictures on the Z1 at roughly 4 second intervals. No problem in my test.

firmware 1.60.1
camera was reset before test tested
after every shot, I have a loop that checks the status of that single shot. When the status is “done”, I move on. You can’t rely on waiting 4 seconds alone. You must check camera status.

The script I used for the testing and the testing tool are here

I’m using a bash script to drive the test.

#!/usr/bin/bash
# tested on WSL2 theta.exe is in the same folder as this script
# endurance test for Z1 with firmware 1.60.1
# prior to test
# * pictures deleted with ./theta.exe deleteAll
# * camera settings were reset with ./theta.exe reset 
# IMPORTANT: The two commands above will delete all your media and settings


# disable offDelay
./theta.exe offDelay --off

# disable sleepDelay
./theta.exe sleepDelay --off

# you may want to have commands in here to delete existing pictures
# and free up camera space. It's not here in the example for safety

# turn off self timer
./theta.exe exposureDelay --seconds=0

# turn off shutter volume "beep"
./theta.exe shutterVolume --low

# confirm settings
./theta.exe getOptions

# set exposureCompensation to -2.0
./theta.exe exposureCompensation --value=-2.0

# take test picture and download to local computer
# it will be in the same directory as theta.exe
./theta.exe takeAndDownload

# set exposureCompensatio to 2.0
./theta.exe exposureCompensation --value=2.0

# take picture and download second test image
./theta.exe takeAndDownload

# set exposureCompensatio to 0
./theta.exe exposureCompensation --value=0.0

# take picture and download second test image
./theta.exe takeAndDownload

# from this point onward, the images are left on camera
# you will need to download the images for inspection

echo start endurance loop test
# example of loop to take  pictures for timelapse or testing
# increase the number below to 300 to take 300 pictures
# example {1..300}
for counter in {1..100}
do
    ./theta.exe takeAndReady
    echo that was picture $counter
    ((counter++))
    # waiting 1 second just in case camera is overheading
    echo waiting 1 seconds for the next shot
    sleep 1s

done
echo finished endurance test

# grab all the thumbnails and write to local storage
./theta.exe download --thumb=all
echo thumbnail download test completed
echo thumbnails are in local storage for inspection

This article might be relevant.

https://theta360developers.github.io/webapi/2020/04/25/camera-status.html

Before you take the next picture, you need to check to make sure the camera status is “done”. You don’t need to download the picture (as the article shows). But, you need to check on the status before you issue another command.

I’m doing another test now and have taken 370 pictures in sequence so far, at an average rate of 4 seconds per shot. I have it set to stop after 500 pictures. I’ll post again after I complete that test.

craig · February 10, 2021, 8:59pm

Second Test

500 shots in sequence

Note that I have a 1 second delay between each shot. I don’t know if the delay is needed. However, as you were having problems, I thought that maybe the camera might overheat. I can try it with less delay between shots if there is a requirement to get the smallest possible delay and 500ms reduction per shot would make a difference.

I believe the firmware can be used. You may have to adjust your loop slightly.

This is the sequence I am using:

take one shot and grab id
call /osc/status and pass id in the body of the post
check if state is done. This may be confusing as there is a /osc/state command with the same name. I am referring to the state key in the response from /osc/status

https://api.ricoh/docs/theta-web-api-v2.1/protocols/commands_status/

if the state is inProgress, then wait 1 second and then check the status again
if the state is done, wait 1 additional second to be safe (you may be able to eliminate this delay)
then take another shot

Repeat steps 1-6 500 times

Pixel · February 10, 2021, 10:12pm

Craig, thank you very much for the quality of your response

I can confirm that my application reacts correctly to the value of the state field (done or inProgress after /osc/commands/execute or /osc/commands/status).

Here are the settings I am using (7 shots - Z1) :
1- ISO 100 - aperture 2.1 - shutter_speed 0.625 - color_temprature 2500
2- ISO 100 - aperture 2.1 - shutter_speed 0.16666666 - color_temprature 2500
3- ISO 100 - aperture 2.1 - shutter_speed 0.04 - color_temprature 2500
4- ISO 100 - aperture 2.1 - shutter_speed 0.01 - color_temprature 2500
5- ISO 100 - aperture 2.1 - shutter_speed 0.0025 - color_temprature 2500
6- ISO 100 - aperture 2.1 - shutter_speed 0.000625 - color_temprature 2500
7- ISO 100 - aperture 2.1 - shutter_speed 0.00015625 - color_temprature 2500

Exemple of sequence (7/10 it’s work):
1- /osc/commands/execute camera.setOptions (shutter_speed: 0.625…) -> done
2- /osc/commands/execute camera.takePicture -> done
3- /osc/commands/execute camera.setOptions (shutter_speed: 0.16666666…) -> done
4- /osc/commands/execute camera.takePicture -> done
3- /osc/commands/execute camera.setOptions (shutter_speed: 0.04…) -> done
4- /osc/commands/execute camera.takePicture -> inProgress (“id”:“333”)
5- wait 4s
6- /osc/commands/status (“id”:“333”) -> inProgress
7- wait 4s
8- /osc/commands/status (“id”:“333”) -> inProgress
9- wait 4s
10- /osc/commands/status (“id”:“333”) -> inProgress
… (Loop: the state is always inProgress)

I just added the call to /osc/state when I exceed the limit (> 10 status inProgress) here is the result I got:

{
    "fingerprint": "FIG_0092",
    "state": {
        "_apiVersion": 2,
        "batteryLevel": 0.76,
        "_batteryState": "disconnect",
        "_cameraError": [
            "NO_DATE_SETTING"
        ],
        "_captureStatus": "idle",
        "_capturedPictures": 0,
        "_compositeShootingElapsedTime": 0,
        "_function": "normal",
        "_latestFileUrl": "http://192.168.1.1/files/90014a6843396150330100a0e99a8700/100RICOH/R0010333.JPG",
        "_mySettingChanged": false,
        "_pluginRunning": false,
        "_pluginWebServer": true,
        "_recordableTime": 0,
        "_recordedTime": 0,
        "storageUri": "http://192.168.1.1/files/90014a6843396150330100a0e99a8700/"
    }
}

To get the error “serviceUnavailable” I have to call /osc/commands/execute…

My impression is as follows:

your tests show that with your equipment (computer and command line tools) the results are very stable
my tests show that with my equipment (ios application - on iOS 14) something is going wrong …
The problem can come from the Payload sent … What surprises me is that my application has been working correctly for 3 years (without the slightest error on thousands of pictures taken).

I may have an explanation: when serializing JSON iOS represents floating point numbers to 15 decimal places, which might go something like: 2.1000000000000001 instead of 2.1… I will test…

craig · February 11, 2021, 12:01am

Hi, thank you for your additional information on the shutterSpeed values. @jcasman and I will try and do a closer replication of your setup over the next few days.

I do not have any additional information on the firmware upgrade.

It’s possible that the firmware upgrade made the API more strict such as what type of float/double number it takes. I do not have any information about this. Although not in the script I shared, I tested the shutterSpeed last week. I believe I accepted a “String” for the shutterSpeed and then converted it to a float/double and then sent the value part of the JSON key/value as a number without quotes.

Sometimes using curl on each of the commands helps to isolate a problem. You could try running the individual commands in curl to confirm what works, then use the XCode debugger to see where it is failing.

If the camera receives an invalid value, I would expect it to return a reasonable error message. However, maybe the long floating point number is causing an unforeseen problem.

I have not tried this, but would the bracket shooting and start capture meet your requirements as well?

@jcasman and I will keep trying to help you isolate the problem. If you find the fix, please post the answer.

It seems tricky to find it.

I’m charging a test iPhone now and I plan to try more testing on iOS in the future.

Pixel · February 11, 2021, 12:50am

Craig, Thanks again for your help

You are right _autoBracket & startCapture are very good alternatives (this change is planned in my road map).

iOS 14 includes many changes, including how access to local network is done, I continue my tests (Floating point numbers FIX) assuming that the problem must be on my side…

I’ll keep you informed !

Pixel · February 11, 2021, 3:14pm

I have the explanation…

The problem is not caused by incorrect formatting of a floating point number (I reproduce the problem also when the formatting is correct).

The problem appears randomly when I place a call immediately after executing a command with the state “done”… I solved the problem by adding a 1 second timeout between each calls (like Craig sample), the result : after several hours of tests, I was able to take 300 photos in a row without any problem (on Theta Z1 & V with the last firmware). Perhaps the best solution is to systematically check the status of the device after execution a command by calling /osc/state…

I think this is an info to send to the RICOH dev team: the device must remain stable whatever the commands received, a simple error message should be sufficient.

Thank’s again for your help!

craig · February 11, 2021, 4:27pm

Congratulations on getting this to work. Thank you for posting the solution. This is useful for many people.

I’ll work with @jcasman to replicate the problem and send it to RICOH. We have direct communication with RICOH management, so although it takes time, there is a method to eventually get the information to the dev team at RICOH.

We can likely replicate it easily by deleting the 1 second wait between each call.

Also, thank you for the idea about using /osc/state instead. We’ll also do more testing on that. Although this may not be relevant in your business, it appears that the SC2 and the Z1 behave differently to /osc/status and /osc/state.

Even though the current problem is solved for your project, @jcasman and I are still planning to test the _autoBracket settings with the shutterSpeed values you provided. I want to see if there is any difference in the time taken for _autoBracket startCapture to complete versus the process described in this discussion.

Thank you again for providing your solution.

Pixel · February 11, 2021, 5:18pm

Great! Do not hesitate to contact me if you need additional information!