You may want to try and approach Jake Kenin to see if he’s interested in a side job to get his application working on the Oculus Go in addition to the Rift. I think it’s totally doable. I believe his application does exactly what you want, but the camera is in a flying drone, not being carried by a human being.
Hugues also has a solution working to a browser, but it conceivable he could port it to a headset assuming you using a browser with WebVR (it’s just a normal browser).