SOLVED: Display smooth live preview in React Native Expo with SDK

I use the Ricoh Theta SDK and try to have smooth render of live preview to a react native app experimentation.

I followed guideline to create react native project with SDK. I’m able to initialize, read live preview and take picture.

Read live preview is not smooth, I mean when I got new byte data, my image is refreshed correctly but the change between the two byte data state is visible in the app.

So I tried react-native demo
and preview is very smooth. I’m not able to do so in my project with documentation

Here is basic code used
I put 2 buttons

  • First to initialize ricoh connection and set up live preview
  • Second to take capture
/**
 * Sample React Native App
 * https://github.com/facebook/react-native
 *
 * @format
 */

import React, { useEffect } from 'react';
import type {PropsWithChildren} from 'react';
import {
  Alert,
  Image,
  Platform,
  Pressable,
  SafeAreaView,
  StyleSheet,
  Text,
  useColorScheme,
  View,
} from 'react-native';

import {
  Colors,
} from 'react-native/Libraries/NewAppScreen';

import {NativeModules, NativeEventEmitter} from 'react-native';
import {
  initialize,

  getPhotoCaptureBuilder,
  BitrateEnum,

  getLivePreview,
  stopLivePreview,

  THETA_EVENT_NAME,
  Options,
} from 'theta-client-react-native';

function App(): React.JSX.Element {
  const isDarkMode = useColorScheme() === 'dark';

  const [dataUrl, setDataUrl] = React.useState<string | undefined>();
  const [takenPhoto, setTakenPhoto] = React.useState<string | undefined>();

  const backgroundStyle = {
    backgroundColor: isDarkMode ? Colors.darker : Colors.lighter,
    flex: 1,
  };

  const startLivePreview = async () => {
    try {
      await getLivePreview();
    } catch(e) {
      Alert.alert('getLivePreview', 'error: \n' + JSON.stringify(e), [
        {text: 'OK'},
      ]);
    }
  };

  const onConnectPress = async () => {

    // http://192.168.1.1
    // https://fake-theta.vercel.app
    try {
      await initialize('http://192.168.1.1');

      const emitter = new NativeEventEmitter(
        NativeModules.ThetaClientReactNative,
      );

      const eventListener = emitter.addListener(THETA_EVENT_NAME, event => {
        setDataUrl(event.data);
      });

      await startLivePreview();
    } catch(e) {
      console.log(e);
    }

  };

  const onTakePress = async () => {
    const photoCapture = await getPhotoCaptureBuilder().build();
    await setOptions({ bitrate: 1048576 });
    const url = await photoCapture.takePicture();

    console.log(url);

    setTakenPhoto(url);
  };

  return (
    <SafeAreaView style={backgroundStyle}>

      <Pressable onPress={onConnectPress} style={[styles.button]}>
        <Text style={[styles.buttonText]}>Connexion</Text>
      </Pressable>

      <Pressable onPress={onTakePress} style={[styles.button]}>
        <Text style={[styles.buttonText]}>Take picture</Text>
      </Pressable>

      {takenPhoto !== undefined ? <Image source={{uri: takenPhoto}} style={[styles.flex]}/> : ''}
      {dataUrl !== undefined ? <Image source={{uri: dataUrl}} style={styles.flex} /> : ''}

    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  flex: {
    width: 300,
    height: 150,
  },

  button: {
    backgroundColor: '#e0e0e0',
    width: 200,
    height: 100,
    marginTop: 50,
    marginLeft: 50,
    flexDirection: 'row',
    alignItems: 'center',
    justifyContent: 'center',
  },

  buttonText: {
    color: 'black',
    fontSize: 20,
  },
});

export default App;

See this issue:

The live preview demo from RICOH had jitters until version 1.6.

Check the commit log and see if you can isolate what RICOH did to make it smooth. Change occurred in the November 2023 timeframe.

from briefly looking at the code, they may have moved it Marzipano in a webview?

image

In the issue discussion, there were other ideas.

Thank you for your reply, my project use Expo so I need to use Ricoh API instead of SDK ? My preview tests were for react native without expo

Hi Wyllis,

funny I am fixing the same issue today! The preview was laggy.

I am able to use Expo with SDK, so should be working correclty. I just needed to use yarn install and not npm to have it installed.

Regarding laggy preview - I believe the problem is with Image component from react-native.

I didn’t want to use marzepan.js as it’s a different UX. But luckily expo has their own Image component from expo-image library.

It’s very similar to react native’s Image component, and after dropping that one in my codebase, the preview is pretty smooth :slight_smile:

There is small delay between what I do and what is displayed in the app, but I believe this is just wifi delay that would be hard to fix.

3 Likes

Thank you arturkesik, I was lost about what should I use. I’m gonna try with yarn as you mentionned but using npm and yarn in the same project are there problem ? I will come back and tell you the result.

2 Likes

Hi arturkesik, craig

After multiple tries I’m able to start project

I generated package from theta-client/react-native

Here is the content
theta-client/react-native/package content

theta-client-react-native installation

I tried a lot of commands to add package with yarn as mentionned by arturkesik

Projects are located at

  • C:/Users/MyName/Documents/projects/theta-client (theta repo)
  • C:/Users/MyName/Documents/projects/react-using-ricoh

Here is commands tried

  • yarn add file:C:/Users/MyName/Documents/projects/theta-client/react-native/package

After I used yarn install

I got the package in my node_modules

node_modules/theta-client-react-native

Build project

I tried npx expo start and got error

Command result

In my project I have a file which need to import theta package so I used this import

  • import { initialize } from 'theta-client-react-native';

VS Code shows me Cannot find module 'theta-client-react-native' or its corresponding type declarations. ts(2307)

After some research

I need to use npx expo run:android instead of npx expo start but some project modification should be made.

  • Execute npx expo run:android to initialize /android folder at the root project (error will apear)
  • Update minSdkVersion to 26 in android/app/build.gradle
  • Now I should always use npx expo run:android to build project (for Android)

I can now build and use SDK

Thank you arturkesik, craig !

2 Likes

This is a great result! Thanks for posting the details @Wyllis_Monteiro and thanks @arturkesik for the great information. Very cool.

1 Like

It was very hard to use the SDK with expo so if this post can help someone else :slight_smile:

2 Likes

thanks to @arturkesik and @Wyllis_Monteiro for providing this information. Is the preview smooth in React Native Expo?

I need to use npx expo run:android instead of npx expo start but some project modification > should be made.

  • Execute npx expo run:android to initialize /android folder at the root project (error will apear)
  • Update minSdkVersion to 26 in android/app/build.gradle
  • Now I should always use npx expo run:android to build project (for Android)

Wow, this is a nice tip. Seems like a hard-earned one. Thanks. :slight_smile:

Hello @craig, the preview is smooth with react native expo, I only tried with Theta X, I will try with SC2 soon. As @arturkesik said we need to use expo-image library. I didn’t try project with iOS device, I will be back for additionnal information about iOS setup if necessary.

Moreover, @arturkesik talked about

There is small delay between what I do and what is displayed in the app, but I believe this is just wifi delay that would be hard to fix.

I’m not able to reproduce that. I don’t know what can can cause this

1 Like

the delay is similar to the one in official theta app - it’s probably just a wifi transmission delay, and taking in account it’s in the official app I believe nothing we can do about it :slight_smile:

I am testing on ios device, so I also needed to install expo-build-properties and configure those in app.json under plugins:

    "plugins": [
      ...
      [
        "expo-build-properties",
        {
          "ios": {
            "deploymentTarget": "16.0",
            "extraPods": [
              {
                "name": "THETAClient",
                "path": "../../theta-client/kotlin-multiplatform/build/cocoapods/publish/debug"
              }
            ]
          }
        }
      ],
      ...
    ],

Glad you got it working :slight_smile:

3 Likes

Thank you for these great tips.

There is going to be an aproximate 350ms delay when transmitting from the camera. The latency is due to the image processing pipeline of the camera, which involves stitching the image, or jpg frame, inside the camera.

do you require lower latency? What is the use case?

2 Likes