WebRtc on THETA X

[coworks_kakuta]

Hello.
I am developing a plugin for live streaming with WebRtc on THETA X.

The following is a sample plug-in for live streaming with WebRtc on THETA V, but I would like to use it as a base for THETA X.

GitHub - ricohapi/theta-plugin-webrtc-sample

The situation is that after launching the plugin and starting streaming, an error occurs and the plugin crashes.
The error is output as “checkAndUpdateEglState: invalid current EGLDisplay”. (Detailed log will be explained later)

Below are the questions.
① I’m having trouble figuring out the cause of the error. Could you please tell me the cause and how to deal with it?
②If you know of any examples of using THETAX with WebRtc in the past, please let us know.

【supplement】

• The sample uses the Android standard camera API (android.hardware.Camera), but this time we want to distribute 360° videos with THETA X, so we changed it to use the THETA version (theta360.hardware.Camera). doing.

【Log when an error occurs】
2024-02-28 16:34:42.806 3203-3456 org.webrtc.Logging com…uginapplication.webrtc.sample I SurfaceTextureHelper: Setting listener to com.theta360.pluginapplication.webrtc.sample.network.WebRTCLibExtends.-$$Lambda$ThetaSession$_GytSbtfY-yp9gmCkhEWH7a1F8k@27846dc
2024-02-28 16:34:42.807 3203-3456 theta360.h…Camera.TAG com…uginapplication.webrtc.sample D handleMessage 8194
2024-02-28 16:34:42.815 3203-3456 theta360.h…Camera.TAG com…uginapplication.webrtc.sample D cameraModeChange state = 1
2024-02-28 16:34:42.816 3203-3520 gst_sensor_src com…uginapplication.webrtc.sample D gst_sensor_poll_event_task has callback, block task, enable_task:0
2024-02-28 16:34:42.834 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_progressed in
2024-02-28 16:34:42.835 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_progressed in
2024-02-28 16:34:42.838 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_progressed in
2024-02-28 16:34:42.845 3203-3521 gst_sensor_src com…uginapplication.webrtc.sample D gst_sensor_poll_event_task has callback, block task, enable_task:0
2024-02-28 16:34:42.849 3203-3522 gst_sensor_src com…uginapplication.webrtc.sample D gst_sensor_poll_event_task has callback, block task, enable_task:0
2024-02-28 16:34:42.945 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_progressed in
2024-02-28 16:34:42.949 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_completed 1
2024-02-28 16:34:42.949 3203-3492 stg_plugin com…uginapplication.webrtc.sample D --ndk_capture_request_get_isHighlight_value-- isHighlight [0] = 0
2024-02-28 16:34:42.949 3203-3492 stg_plugin com…uginapplication.webrtc.sample D color_temp is 0, cameraID = 0, iso = 226, ex_time = 30000000
2024-02-28 16:34:42.964 3203-3523 stg_plugin com…uginapplication.webrtc.sample D camerabin_image_reader on_image_available in, is ready 0
2024-02-28 16:34:42.964 3203-3523 stg_plugin com…uginapplication.webrtc.sample D camerabin_image_reader on_image_available long_ex_image = 0, self->shutter_begin = 0
2024-02-28 16:34:42.964 3203-3523 camerabin_src com…uginapplication.webrtc.sample D camerabin_src gst_camera_source_on_ndk_image_preview 1042: in
2024-02-28 16:34:42.964 3203-3523 stg_plugin com…uginapplication.webrtc.sample D stg_dual_image_new called
2024-02-28 16:34:42.964 3203-3523 stg_plugin com…uginapplication.webrtc.sample D image->refcount 1
2024-02-28 16:34:42.964 3203-3523 stg_plugin com…uginapplication.webrtc.sample D gst_camera_sync_post_bus_message ret = 1 message = first-frame-done
2024-02-28 16:34:42.971 3203-3523 stg_plugin com…uginapplication.webrtc.sample D gstdisplay->nwBuffer = 0x79a2b49bd0
2024-02-28 16:34:42.972 3203-3523 gst_display_sink com…uginapplication.webrtc.sample D display nwBuffer out = 0x79a2b49bd0
2024-02-28 16:34:42.978 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_progressed in
2024-02-28 16:34:42.980 3203-3455 theta360.h…Camera.TAG com…uginapplication.webrtc.sample D postEventFromNative->16
2024-02-28 16:34:42.981 3203-3523 gst_display_sink com…uginapplication.webrtc.sample D gst_displaysink_chain Entry
2024-02-28 16:34:42.981 3203-3523 gst_display_sink com…uginapplication.webrtc.sample D display map_out ANativeWindowBuffer 0x79a2b49bd0
2024-02-28 16:34:42.981 3203-3523 gst_display_sink com…uginapplication.webrtc.sample D gst_displaysink_chain Exit
2024-02-28 16:34:42.981 3203-3456 theta360.h…Camera.TAG com…uginapplication.webrtc.sample D handleMessage 16
2024-02-28 16:34:42.981 3203-3523 stg_plugin com…uginapplication.webrtc.sample D stg_dual_image_unref called
2024-02-28 16:34:42.981 3203-3523 stg_plugin com…uginapplication.webrtc.sample D image->refcount 2
2024-02-28 16:34:42.982 3203-3523 stg_plugin com…uginapplication.webrtc.sample D stg_dual_image_unref called
2024-02-28 16:34:42.982 3203-3523 stg_plugin com…uginapplication.webrtc.sample D image->refcount 1
2024-02-28 16:34:42.982 3203-3523 stg_plugin com…uginapplication.webrtc.sample D g_free called
2024-02-28 16:34:42.982 3203-3523 stg_plugin com…uginapplication.webrtc.sample D image->refcount 0
2024-02-28 16:34:42.982 3203-3456 OpenGLRenderer com…uginapplication.webrtc.sample E [SurfaceTexture-1-3203-1] checkAndUpdateEglState: invalid current EGLDisplay
2024-02-28 16:34:42.984 3203-3492 stg_plugin com…uginapplication.webrtc.sample D camerabin_session on_capture_completed 1
2024-02-28 16:34:42.984 3203-3492 stg_plugin com…uginapplication.webrtc.sample D --ndk_capture_request_get_isHighlight_value-- isHighlight [0] = 0
--------- beginning of crash
2024-02-28 16:34:42.984 3203-3492 stg_plugin com…uginapplication.webrtc.sample D color_temp is 0, cameraID = 0, iso = 226, ex_time = 30000000
2024-02-28 16:34:42.993 3203-3456 AndroidRuntime com…uginapplication.webrtc.sample E FATAL EXCEPTION: VideoCapturerThread
Process: com.theta360.pluginapplication.webrtc.sample, PID: 3203
java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)

2024-02-28 16:34:43.001 3203-3523 stg_plugin com…uginapplication.webrtc.sample D camerabin_image_reader on_image_available in, is ready 0
2024-02-28 16:34:43.001 3203-3523 stg_plugin com…uginapplication.webrtc.sample D camerabin_image_reader on_image_available long_ex_image = 0, self->shutter_begin = 0
2024-02-28 16:34:43.001 3203-3523 camerabin_src com…uginapplication.webrtc.sample D camerabin_src gst_camera_source_on_ndk_image_preview 1042: in
2024-02-28 16:34:43.001 3203-3523 stg_plugin com…uginapplication.webrtc.sample D stg_dual_image_new called

[craig]

This one runs on the X.

GitHub - ricohapi/theta-wireless-live-streaming-plugin: Wireless live...

Long ago, @biviel started with that and I think he may have added WebRTC to it.

This one works with RTSP, but not the X.

GitHub - science-arts/THETA-RTSP-plugin: RTSP plugin for THETA

Another possibility is to start with this:

GitHub - ricohapi/theta-plugin-camera-api-sample-x: THETA X Plug-in : Camera...

At the moment, I am having problems building the theta-plugin-camera-api-sample-x, but I did build and run it a month ago. I’m not sure how I edited the gradle files before.

[coworks_kakuta]

Thank you for your reply.

“theta-wireless-live-streaming-plugin” and “theta-plugin-camera-api-sample-x” are already referenced.
I haven’t seen “THETA-RTSP-plugin” yet, so I’ll use it as a reference.

Also, do you have any idea of the main cause or countermeasure for the error “checkAndUpdateEglState: invalid current EGLDisplay” that I wrote about earlier?

[craig]

@coworks_kakuta

Also, do you have any idea of the main cause or countermeasure for the error “checkAndUpdateEglState: invalid current EGLDisplay” that I wrote about earlier?

I have a similar error on the theta-plugin-camera-api-sample-x. If you can get the same error, then you can open an issue on the theta-plug-camera-api-sample-x and hopefully open a dialogue on GitHub issues with the RICOH engineers.

[craig]

Another option is to try and use this web form and ask about the error message you are getting. Maybe the support can help?

RICOH THETA Plug-in Partner Program｜RICOH

Also, remember that the THETA X is running Android. So, maybe you could search for an open source Android application project for WebRTC and use that as a base for the THETA X plugin?

[coworks_kakuta]

Thank you for your reply

I have a similar error on the theta-plugin-camera-api-sample-x.

I checked in my environment and this error did not occur with “theta-plugin-camera-api-sample-x”.

Another option is to try and use this web form and ask about the error message you are getting. Maybe the support can help?

I have contacted the help desk regarding several other matters, but they have been left unattended for over a month and I can’t rely on them.

[craig]

In the article below, the author of the article on Qiita and the author of the code theta-plugin-webrtc-sample appear to be the same. You could try and open an issue on the theta-plugin-webrtc-sample repository and also comment on the article on Qiita.

Create a WebRTC P2P Shooting App with the THETA Plug-in

This article was originally published in Japanese by Hideki Shiro (@shrhdk_ on Qiita). Introduction Hello, @shrhdk_ of Ricoh. I will introduce a plug-in that uses WebRTC to trigger the shutter of a RICOH THETA V while remotely checking the 360° image of RICOH THETA V in real time. The plug-in is a result of trying to build a web-based application that provided the remote shutter function of the THETA standard application using WebRTC only. [graphic-1] Source Code What is WebRTC? First…

This article uses the Skyway SDK.

THETA Live Streaming 360° Video with WebRTC Direct From Camera

This article was originally published on Qiita in Japanese by masasutzu. I tried out SkyWay - Android - SDK 's p2p - videochat on interesting things whether it will work on THETA V. As a result, I was able to run p2p-videochat on THETA V, and flowed 360 degrees video to the other party. I will explain in detail later in this article, but I told gogoroya that I was frustrated at first. I am able to confirm that I can play audio / video by doing SkyWay-Android-SDK with THETAPlugin. #skyway # …

This old one used OpenTok

Making Theta 360 work with OpenTok's client-side webRTC (livestreaming - mobile stitching - google "cardboard" view)

Hi there, just wanted to make a post on how discuss.io forked the theta_gl repository to enable 360 functionality in the desktop (firefox/chrome) and mobile (android only) browsers. I’ve spent the last month building this thing and wanted to thank everyone on the forum for letting me lurk and learn We recently presented it at conferences around the world and our customers are excited about using this technology to observe their customers in the real world! You can find the PR he…

This one also for the V

AI Assisted 360° Live View for Theta V / Z1 Cameras by Attila Tőkés

A Theta 360 plugin for providing live view over WebRTC, assisted with TensorFlow based object detection. [image] Story Introduction In this project I will show how I built the AI Assisted 360° Live View plugin for the Theta V camera. The plugin provides 360° Live View functionality using WebRTC , and object detection using TensorFlow . In a more advanced form, the plugin could be used to remotely assist personnel in different use cases , like for example in rescue operati…

---

Are you using the X because you want to use the large LCD screen of the X?

[coworks_kakuta]

Thank you for all the information.
I will check it.

Are you using the X because you want to use the large LCD screen of the X?

I don’t need an LCD screen, but the request specifies X, and I only have X.