"HDR Wireless Live Streaming" Plugin - New features soon

THETA API Usage plugin
1

I’m preparing a huge (SUMMER edition) update for the “HDR Wireless Live Streaming” plugin, don’t confuse it with Ricohs “Wireless Live Streaming” plugin, which is very limited in quality and performance, please.

  1. Optimized for summer environment conditions, so that it would operate much longer compared to previous version(which is now there alredy in plugin store). There will be a switch to turn on or off this optimization, even directly by pushing buttons on Z1, or via flow.tours configuration page. Plugin will automatically lower slightly the average FPS if camera get’s too hot. By optimizing peak temperature, I could duplicate the operation time even above 25°C. Different level of optimizations will be there to preset.

  2. Changing exposure compensation and framerate during live stream or recording - by pushing buttons on Z1, or via flow.tours configuration page directly

  3. Video effects can be applied in camera, like turning live stream or recording to greyscale, or apply emboss, edge detection effect, lower saturation effect

  4. Upload your own nadir logo - through flow.tours you will be able to directly upload your own image to place as nadir at the bottom, so it will be visible when you stream next time.

  5. Microphone volume/gain control - the volume of the internal microphone in Z1 was too low, so I built a gain control, you will be able to BOOST and make streams or recordings much louder, without increasing the noise, by normalizing it at the same time.

  6. Remotely start or stop streaming - after this plugin update is done, through flow.tours configuration page you will be able to set your Z1 to automatically start streaming, or to stop streaming remotely. So you can actually use your Z1 for remote surveillance too.

  7. Streaming to OBS Studio or vMix desktop tools as source via WiFi - for some people it’s important to stream via OBS or vMix as “source”, like a webcam, so yes, when Z1 is connected to the same network as desktop PC, via WiFi, will be able to stream into these directly, use multiple sources, or more of them too.Also this will work in HDR video mode, as well as using H.265 encoding and using LOW bitrate. I noticed that some Theta Z1 users were complaining about high bandwidth usage when Z1 is connected to a desktop with USB C cable or LAN… This will be under control.

  8. SRT protocol support, will make this new protocol finally public, so you will be able to stream to your SRT servers directly.

  9. Adaptive bitrate during live stream - I put this to the list of features that I will be able to deploy. According to plan, there will be 2 types. 1. Default type - it will automatically lower bitrate and quality of the stream, but keep FPS, 2. Variable type - if plugin detects loss in bandwidth , will automatically lower the bitrate, but also will lower FPS at the same time, so the goal is that image quality will remain the same, while FPS will be lowered as long as bandwidth is not good enough.

  10. Fast/Seamless Roaming in WiFi mesh network - as an outcome Theta Z1 is THE ONLY 360 camera for live streaming that can switch from one WiFi router to another in less than 50ms (ideally less than a duration of a single frame capture, but there can be some networks and situations when it’s noticeable). So it will be barely noticeable by viewers of the live stream. Technical details related to Fast Roaming: WiFi Fast Roaming, Simplified | Network Computing .
    Roaming works even without my plugin, but it’s not so fast and during tests I did it was causing a big interruption during live streaming. With this feature a large area can be covered and walk during live streaming, there can be put multiple mesh networked routers, I tested this with these: https://www.amazon.com/TP-Link-Deco-Whole-Home-System/dp/B07NF6V8SC LAN internet plugged in to first “master” router and it shared internet through wifi with rest of the devices, or by LAN. Can connect more than 3 routers, 5-6 of them, for museums, factories or larger events, halls, it will make sure that a stable 20-25mbps connection will work superb and switch over when closing to a stronger signal, seamlessly, undetected to viewers. I’m going to expose a setting for this plugin, so users can set the peak signal level when Theta Z1 should initiate to roam over to next AP unit in the mesh network. If there is no mesh network at all, it can cause coverage issues, or if it’s not properly setup. With a mobile phone there are tools to check wifi signal strength across the room/environment.

If you have any thoughts or questions, let me know! And yes, I’m still pushing flow.tours platform too, so soon you should be able to see or even try it out to stream to. For now you will be able to try above features soon!

3 Likes
2

This is a great plug-in. I put this out on the .guide Twitter feed and Facebook Page.

The heat mitigation strategy sounds wonderful.

Nice add with the volume boost control.

The latency should be much lower than YouTube. Are you going to launch the flow.tours platform in the US when you launch the plugin.

What is your strategy for “save-to-file”, especially the ability to save to file longer than 25 minutes? For the save-to-file, is there a way to adjust the framerate and bitrate?

1 Like
3

Thanks, Craig!

It will require some finetuning, it will have lot of hidden options initially, like I defined 2 peak temperatures for internal CPU/board , by default first at around 52.5°C and second level above 56°C, also measuring how fast does it change, etc. dynamically increasing ot decreasing temperatures. It was interesting the first time to se that decreasing FPS for 5-10 minutes and then increasing back to 24 helps much more than those 5-10 minutes… Like it was then streaming 24FPS for ~20 minutes extra. I’m making all these temperatures, counters, measurement frequency, differentials configurable remotely, so I can create later 2-3 different profiles if needed, or let the user adjust via UI. Some switch will be also available by Z1 buttons directly to turn on/off these optimizations.

I was able to further decrease(stabilize) latency and increase quality, I’m excited about this. I’m looking at different server regions across US right now to pick the best option. I may reach out to you directly about this if you have some time. I will launch the new plugin first and will let people use my platform step by step as still lot of questions about various hardware devices, etc. I’m planning to make it available slowly.

For save-to-file, all applies as for live streaming: 1. heat optimizations, 2. video effects, exposure changes, FPS manual adjustments, start and stop remotely, etc.

Next phase could be some AI and object detection integration, running in camera… But pushing platform would be more important at this point.

1 Like
4

I added a new feature to above list that I can confirm will be able to implement and make streams much smoother in various network conditions. Adaptive bitrate during live stream. See point 9 above for details.

1 Like
5

The technical features are outstanding.

You should build a market entry launch plan.

What are the published service fees going to be to use the plug-in with Flow Tours?

Are you planning to use resellers in the US and other parts of Europe and Asia?

Maybe you could start by approaching photographers like Toyo to see if there’s a way to get a local representative for Flow Tours to provide the service? You’d need to figure out a discount model for resellers to make sure the resellers make money.

If you post on Facebook and other groups, you may be able to find 360 photographers and videographers in the US that want to branch out into 360 live streaming services.

The important thing is to make sure that your business partners (resellers) make money as you make money.

If they already have a few Z1s or at least 1, they can get started easily.

A different model is charge a franchise fee, but you generally need an established business going at one location first.

1 Like
6

Will adaptive bitrate also lower the heat of the live streaming session?

Similar to fps, maybe an additional benefit of adaptive bitrate is improved heat management?

7

When “Variable type” adaptive bitrate is used, it can lower cpu heat too, but mainly it will be activated only if the desired video bitrate is not possible through actual network conditions. BUT adaptive bitrate mode will work together with “heat optimization” feature, so both can be turned on at the same time. Clearly I I need to see which will take priority when both are about to lower FPS, I think it should pick the lowest FPS value in that case. if proccess 1 (which is adaptive bitrate process) says, lower FPS to 20, but heat optimization process will say set FPS to 22, actual fps will be set to 20. So the lowest would be picked. If we see a good network connection so adaptive bitrate process says put fps to 24, but heat optimization process says 22, it will be set to 22. At least this is my plan to implement, not sure yet, for this I may need more details and tests, not sure if it will be done in this next release.

thanks, Craig, I was in need for these features. :slight_smile:

That’s why I’m careful about for now, because I do not see it through yet, which way to go with it. Bottom up was my plan initially, at least for one use case/demo, through flow.tours

This is the hardest question, I will see once I’m able to start a minimized, limited capacity live streaming service online. May be it will work for a month and will be able to see total cost. AWS services are not easy to calculate as it highly depends on CPU usage and downloaded/transferred bytes too, we can’t talk about caching strategy in case of Webrtc for now, at least in my case not->very low latency and high bandwidth, high quality video.

for some use cases yes, will probably only work with resellers, who will make money from it on different way. The most typical is to have an event and someone pay for the reseller to live stream the event clearly, now there could be different options here too: public event or restricted/commercial events where tickets are being sold. For “resellers” probably a low cost subscription based fee will work, which grants monthly X hours of streaming time, which need to book in advance. Clearly at the beginning there will be 1 server, with preset capacity, but the architecture will go beyond that if there is business need for it, will see.

exactly, that’s key, I only want to make money if “resellers” are able to make money with it.

I’m trying to look around also for those who are interested in live streaming, not necessarily 360, but to convince them about 360 technology to try

I do not understand this approach at the moment, my brain is low a bit right now, I may need to watch The Mandalorian tv series now :slight_smile:

1 Like
8

one more feature added above:

10. Fast/Seamless Roaming in WiFi mesh network

9

@biviel Does this mean that the stream can transition from node to node within the mesh network? Are Wi-Fi mesh networks used in public institutions (like museums) that you’re targeting?

1 Like
10

Yes, it means that and in theory in most cases it can transit now without interruption. The issue is that at museums and other places, placement of these wifi routers isnt optimized for live streaming to cover full area, so it interrupts when walking at “blind spots”. Those blind spots are failing only when someone is trying to consume 15-25mbps, but for non livestreaming purposes it would still work fine.

I also finetuned this by adaptive bitrate on wifi too, measuring actual signal strength and when z1 is far from actual router im lowering bitrate and/or fps, so it will extend the streaming range of that router. There are multiple peak levels defined, but i will let that adjustable for pro users by remote configuration file. Combining this adaptive bitrate based on wifi signal strength with fast roaming feature should help a lot to extend coverage. I will be able to finetune these settings later remotelly to each device once i get more test data…

1 Like
11

The technology sounds fantastic. You should try and get either a partner in New York or Los Angeles or figure out a way to get representation.

Los Angeles has many film, music and art schools. If you have any budget available, you could attempt to hire an intern to introduce the system to museums and other types of entertainment venues.

I’m just expressing my personal opinion. As the owner of the business, you need to make your own decisions.

Your technology likely has an edge, but I think you need more focus on sales and marketing. My personal opinion is that a presence in the US, even a part-time person would help. Of course, it’s risky and you could lose your investment in the person’s pay and not see any sales results.

2 Likes
12

hi, @craig ,
yes, I will do so, I could hire my cousin to help me there, after giving her some remote training. :slight_smile:

I appreciate your personal opinion for sure. Indeed, I’m still focusing too much on technology improvements. I decided 2-3 weeks back to push a bit more and collect all potential technical gaps, that’s why I decided to push a bit more plugin development. After the tests I did indoor through WiFi or outdoor and after seeing some results of people who were using it to live stream I collected these requirements.

2 Likes
13

Your cousin might be a good start as she could hold onto the cameras and help introduce people to the system. There’s many students studying film in the Los Angeles area. They might be interested in exploring the 360 video way of streaming live experiences.

1 Like
14

just wanted to give you a quick update. I’m still working on the feature list. There is one very important one I’m finalizing now: adaptive bitrate. I would like to share a few words on this. My goal is to cover it at both protocols: TCP (used by RTMP, RTSP) and UDP (used at SRT, WebRTC, etc).

The most simple approach is: we measure loss of frames, or lost packets, packets dropped in period of time and we decrease the bitrate accordingly. This sounds simple but after you try to implement it’s not so simple:

  1. Z1 connects through WiFi to a mobile phone or cellular hotspot.
  2. Both Z1 and hotspot/mobile phone may be moving, so there is a network connection between z1 and mobile phone or hotspot too, that can potentially drop frames and loose packets too, not only because of the bandwidth changes on cellular network.
  3. Need to do simple math, it must run inside Z1 while live streaming at the same time.

Realistic goal:

  • if someone measures available bandwidth and sets the video bitrate accordingly, it should provide much smoother live streaming, minimizing interruptions compared to live streaming without adaptive bitrate.
  • For 360 live streaming, we should predefine a bitrate range and make adaptive bitrate work accordingly:8mbps-25mbps. If a user sets video bitrate to 18mbps, when adaptive bitrate is set, it would still be required the bandwidth not to drop below 8mbps, otherwise live streaming under 8mbps and below makes no sense. Also at platforms I tested, doesn’t make sense going above 25mbps at all, because for viewing it at viewers end, will not provide much in quality going beyond 25mbs.
  • An optimal solution would be to avoid packet loss during live streaming, but that would require going “safe” and lowering actual bitrate below available bandwidth by more than 25%. Isn’t that a waste of bandwidth and quality? With “simple” approach , by measuring only packet loss, we can’t make make this work, we need way more complex approach: measuring bytes sent in period of time constantly as often as possible and to calculate statistics of available buffer and it’s actual size. This will give us a good hint, especially if we try to predict it for next 24-48 frames. Because of the nature of how live streaming works, we can’t do “optimal approach” without predictions, which will again cost us CPU time and will generate heat. I still think this will work and minimize interruptions.

Work in progress…

1 Like