VR Media Connection BETA Documentation - ENGLISH


#1

360° Browsing Made Easy Combining Oculus Go and RICOH THETA V

Introduction

Hello, this is @roohii_3 from RICOH.

Using the plug-in functionality of RICOH THETA, we’ve made a plug-in to view THETA files directly in an Oculus Go.

Previously, we’ve introduced a way to view from the Oculus browser. With this plug-in, files can be viewed using the Oculus Gallery app.

If DLNA-enabled, there is a possibility to view from devices other than Oculus Go. For example, it has been confirmed that THETA files can be viewed from the Media Player in PSVR+PS4 Pro.

Connect THETA and Oculus Go with Wi-Fi, and hold THETA’s Mode button. RICOH THETA is displayed in Oculus gallery like this.

Select the desired still image or video from the thumbnail and a 360° display will appear.

If the 360° display does not appear, set 360° display to projection setting.

The Plug-in (beta version) introduced in this article can be installed from the link below. If you have RICOH THETA V and Oculus Go, please try it.

This is a beta version, so there is no guarantee that this works perfectly.

https://pluginstore.theta360.com/plugins/com.theta360.vrmediaconnection/

Please see here for plug-in install method and modification method.

Usage

There are 2 ways to set up and use the plug-in.

[1] Using THETA and Oculus Go in a 1 to 1 connection.

When the THETA is set to Access Point (AP) Mode, the THETA and Oculus Go will be directly connected for viewing.

Steps

  1. Press the THETA wireless button and set to AP Mode (LED will flash blue)

  2. Connect to THETA from Oculus’ wireless LAN setting

  3. Press the THETA Mode button, and set to Plug-in Mode

  4. Select Oculus gallery, and select “RICOH THETA”

When watching a video, there might be some skipping. From your smartphone, I suggest setting the THETA’s Wi-Fi frequency band to 5GHz.

*When setting the frequency band to 5GHz, please use it indoors only.

[2] When using THETA and Oculus Go on the same Wi-Fi network

This is when setting THETA in Client (CL) mode, and connecting a THETA and an Oculus Go to the same Wi-Fi network. It is also possible to see THETA files from multiple Oculus at the same time.

Advanced Preparation

  1. With the smartphone app, set the password and SSID for the Wi-Fi network which will be connected to the THETA

Setup Configuration (YouTube), Manual

Steps

1-1. Press the THETA’s wireless button and set it to CL mode (LED will flash green)

1-2. The wireless LED will change from flashing to lit, confirming the THETA is connected to the Wi-Fi network

  1. From Oculus’s Wi-Fi network setting, connect to the same Wi-Fi network as the THETA

  2. Hold the THETA Mode button, and set to Plug-in Mode

  3. Select the Oculus Gallery, and select “RICOH THETA”

Cautions

  • This is a beta version. There is no guarantee of perfect performance.

  • Video performance depends on the Wi-Fi network environment, and it could be disrupted. This issue might be improved by setting the bitrate to Low in the THETA’s video recording setting or by using 5GHz wireless LAN.

  • Both [1] and [2] can be used for Oculus Go, but in some machines, [1] cannot be used. [1] cannot be used for PSVR, so please try [2].

  • In Default setting, video is saved in Dual-Fisheye format. From the smartphone basic app, turn video’s stitch during shooting to ON, then video will be saved in a format to display 360° in Oculus (Equirectangular format).

  • During viewing, there is no zenith correction. We suggest that THETA be set to to stand straight up when shooting.

  • Oculus Gallery has a download menu, but you cannot download while in AP Mode.

About RICOH THETA plug-ins

For people wondering what RICOH THETA is, it is our company’s 360° camera. THETA V is the newest model in the THETA model series and functions with Android. “RICOH THETA plug-ins” can customize the THETA, and you can create them the same way you create Android apps.

There are more articles of plug-ins in RICOH THETA plug-in development community. Please join if you are interested.

Please see this link for more detailed information on plug-ins. Please follow Twitter and participate in THETA plug-in developer community.

Actual Usage

Here is how you use the plug-in.

This plug-in is implemented following the Digital Media Server (DMS) functionality of DLNA. DMS is a function which delivers video and music to other DLNA compatible machines. THETA becomes a server delivering the content, and Oculus Go becomes a client viewing the content.

  • This plug-in does not implement all the functionality of DMS. Please be aware that there is no guarantee that all DLNA・DMS functions will work

I am still learning about this, so I cannot explain it perfectly well, but it seems that DLNA provides the base for a protocol to connect Universal Plug and Play (UpnP) devices to each other. XML is used for an interchange of information between machines, but it seems HTTP is what delivers the actual content.

Cling

I’ve left complicated processing around UPnP - DLNA to a library called Cling. Recognition of devices on the network and exchange of information between devices was mostly handled by Cling. Cling was also responsible for the generation of XML but in fact did not touch XML directly.

For implementation, I referred to a chapter in the Cling Core manual "5. Cling on Android.” To create a media server like this plug-in, refer to "3. Accessing and providing MediaServers" in the Cling Support manual.

Device Information

"5.3. Creating a UPnP device" in the Cling Core manual contains an example of implementation of UPnP service. If you used it for this plug-in, it would look like the code below. In addition, XML for the device information is automatically generated according to the description in this code. The generated XML is sent to other DLNA devices, and the basic information and service information for THETA is recognized.

To briefly explain the contents of the code, in the beginning of the code, basic information on the device equivalent to the media server (THETA in this example) is configured.

In the middle of the code, we are defining services to link with devices. For one device, set one or more services according to the function needed. In the case of a media server, services like "connection manager" and "content directory" are necessary.

At the end of the code, we link the basic information and services we defined to the device.

protected LocalDevice createDevice()
        throws ValidationException, LocalServiceBindingException {

    // Define basic information for the device
    DeviceType type = new UDADeviceType("MediaServer", 1);
    DeviceDetails details = new DeviceDetails(
            "RICOH THETA",
            new ManufacturerDetails("RICOH"),
            new ModelDetails(
                    "VR Media Connection BETA",
                    "VR Media Connection for RICOH THETA",
                    "0.1.0"
            )
    );

    // Define service (function)
    LocalService contentDirectory = new AnnotationLocalServiceBinder()
            .read(ContentDirectoryService.class);
    contentDirectory.setManager(
            new DefaultServiceManager<ContentDirectoryService>(
                    contentDirectory, ContentDirectoryService.class));

    LocalService connectionManager = new AnnotationLocalServiceBinder()
            .read(ConnectionManagerService.class);
    connectionManager.setManager(
            new DefaultServiceManager<ConnectionManagerService>(
                    connectionManager, null) {
                @Override
                protected ConnectionManagerService createServiceInstance() throws Exception {
                    return new ConnectionManagerService(sourceProtocols, null);
                }
            }
    );

    // Load icons
    Bitmap icon = BitmapFactory.decodeResource(getResources(), R.drawable.theta_image_01);
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    icon.compress(Bitmap.CompressFormat.PNG, 100, baos);

    // Link those defined above to LocalDevice
    return new LocalDevice(
            new DeviceIdentity(udn),
            type,
            details,
            new Icon("image/png",
                    icon.getWidth(), icon.getHeight(), 8,
                    "icon.png", baos.toByteArray()),
            new LocalService[]{
                    connectionManager, contentDirectory});
}

Generation of Content Information

XML is also used to convey content information to the client side.

Contents handled by THETA are video and still images, but image information has the following format. It is called "DIDL - Lite" format and seems to be defined by UPnP.

(File Title) (Creator) object.item.imageItem (Thumbnail URL) (Content URL)

The above xml is automatically generated when you store content information in Cling’s Item class as shown below.

String MIMETYPE_JPEG = "image/jpeg";

Res res = new Res(new MimeType(
        MIMETYPE_JPEG.substring(0, MIMETYPE_JPEG.indexOf('/')),
        MIMETYPE_JPEG.substring(MIMETYPE_JPEG.indexOf('/') + 1)),
        filesize,
        fileUrl);
res.setResolution(width, height);

ImageItem imageItem = new ImageItem("id", "parentId", "title", "creatorName", res);
imageItem.addProperty(new DIDLObject.Property.UPNP.ALBUM_ART_URI(new URI(thumbnailUrl)));
imageItem.setRestricted(true);

Directory information is managed in Container class, and information such as movies and still images is managed by Item class. Item seems to have several derived classes specialized for content such as ImageItem and VideoItem.

Set an arbitrary character string for the content ID (id) and parent content ID (parentId) in the above XML and code.

The term "parent content" refers to "directory" here, but the directory and the content under it are linked with these two IDs as a set. This creates a hierarchical structure of content. It seems that it is necessary to set the parent content ID of the root Container to "- 1". [From "UPnP specification" "ContentDirectory: 4" > "5.2.13 Hierarchical location" item (p.41)]

Although it may be unnecessary if I am able to use Cling, I prepared a separate class "Container and Item structure managed by ‘Content ID’".

Content Distribution

It seems that it is necessary to implement ContentDirectory on the media server side as shown in the manual. According to the manual example, this plugin also created a class that inherits AbstractContentDirectoryService. Putting up something using browse() of this class looks like the following code.

    @Override
    public BrowseResult browse(String objectID, BrowseFlag browseFlag,
                            String filter,
                            long firstResult, long maxResults,
                            SortCriterion[] orderby) throws ContentDirectoryException {

    try {
        DIDLContent didl = new DIDLContent();

        DIDLObject content = ContentTree.getNode(objectID).getContent();

        if (content instanceof Item) {
                didl.addItem((Item) content);
                return new BrowseResult(new DIDLParser().generate(didl), 1, 1);

        } else {
                for (Container container : ((Container) content).getContainers()) {
                didl.addContainer(container);
                }
                for (Item item : ((Container) content).getItems()) {
                didl.addItem(item);
                }

                String xml = new DIDLParser().generate(didl);
                return new BrowseResult(xml,
                        ((Container) content).getChildCount(),
                        ((Container) content).getChildCount());
        }

    } catch (Exception ex) {
        throw new ContentDirectoryException(
                ContentDirectoryErrorCode.CANNOT_PROCESS,
                ex.toString()
        );
    }
}

The ContentTree appearing in the code is the "class that manages the structure of Container and Item with the ‘Content ID’” mentioned in the "Generating Content Information" section. Using the content ID as a key, the corresponding DIDLObject (Container and Item are derived classes of DIDLObject) is retrieved.

The objectID of the argument for browse() is the same value as "content ID", so that the appropriate DIDLObject is retrieved from ContentTree.

The extracted DIDLObject is converted to DIDL-Lite format XML by DIDLContent by addContainer() or addItem() and DIDLParser().generate().

This browse() seems to be called "when the client first accesses the server" and then "every time the client accesses the underlying directory.”

The content ID corresponding to the client’s selection is sent as the argument objectID, but when accessing for the first time it seems that the content ID of the root Container is specified. (Probably, I think that it is judged that the parent content ID is set to “-1”.) [From “UPnP specification” “ContentDirectory: 4” > “5.2.13 Hierarchical location” item (p.41)] If you can get the root Container, you can use the content ID - parent content ID to stick together the lower layer you can see in the directory.)

The client side receives content information in DIDL-Lite format via browse(), and shows the received information in the player, etc.

HTTP is used when distributing individual still images and videos.

The value of fileUrl is set to Res in the code of the "Generate content information" section, but if there is a request for content distribution from the client, it will be delivered via this URL.

NanoHTTPD is used in this plug-in. When a content distribution request is made, it has been implemented to read the file corresponding to the specified URL and return it as the result.

Conclusion

Since this plug-in is still in beta, it may have some usability issues or other problems. I am planning to improve it on a daily basis leading up to the official version release!


#2

I ran some additional tests with Linux using VLC. It works fantastic on Linux.