MixCast VR and Intel RealSense Technology

How do you describe what a color smells like?

Virtual reality has been the next big thing for quite a few years at this point, slowly growing in popularity, with tech journalists regularly declaring that VR is dead and never going to be popular. VR is an interesting technology in that great VR experiences can be magical and transformative, and impossible to describe to someone else. If you have ever tried to explain a compelling dream to someone else, or tried to describe what a color smells like, you will understand the difficulty of explaining a great VR experience to someone else.

For Virtual Reality creators, arcade owners and headset manufacturers, this has been an ongoing problem. Showing solely what the player sees can give the impression that VR is just another video game or movie, as well as being quite disorienting, as using the player’s viewpoint as a camera results in a very fast moving and confusing camera. It’s possible to show a VR game from a virtual camera viewpoint too, but again, this still gives any recording a feeling of “this is just a video game.” Without the ability to accurately and effectively show or describe virtual reality, adoption is slow, as enthusiasm for the product generally occurs primarily after trying VR, something that inherently limits the rate of adoption.

Mixed Reality capture

The solution to this communication problem is known as Mixed Reality capture. Not to be confused with Mixed Reality, the term used by some in the industry to refer to devices or experiences that include virtual or augmented reality elements. Mixed Reality capture involves filming a person inside a VR experience and simultaneously recording the virtual elements of the scene. Once this footage has been captured, the two elements are blended together, giving you a film of a real human interacting with a virtual world. This mixed reality footage demonstrates much more clearly to an observer what a VR experience is actually like, engaging them in the content in a way that would only be otherwise possible with a direct VR experience.

How does it work?

Mixed reality capture requires a few different things to allow for an accurate representation of the experience. There are also a number of technical challenges to overcome. Firstly, the real camera that is filming the participant must have an identical digital twin virtual camera within the game, from the same angle and position in the virtual scene as the real camera. If the two are not aligned correctly, the resulting end footage won’t appear as if the player is interacting with the virtual environment correctly.

Similarly, if the player is using controllers to interact with virtual objects, where those controllers appear in the live footage and where they are in the virtual world has to be perfectly aligned.

The other important factor here is that the player’s real environment has to be removed from the footage so that they can be correctly placed in the virtual environment. This can be achieved by a few different methods, such as a traditional greenscreen, but this can be difficult and time consuming to set up, and unlike with traditional filming, since VR content surrounds the player, there may be moments when capture makes more sense from a different angle – which would require an entire greenscreen room.  There are other reasons why greenscreen capture can be undesirable, for example in a VR arcade which will see a lot of traffic greenscreen environments require a lot of maintenance, for example, painting over scuffs or marks on the floors and walls. Greenscreen rooms are also not particularly cosmetically appealing environments. For these reasons and more, using a depth camera to create mixed reality footage can be a better solution. Depth cameras don’t require the greenscreen to segment someone out from their background, and can also better handle occlusion – helping a system to understand what virtual objects should be in front of the player and which should be behind them.

MixCast – the full solution

Blueprint Reality has created the MixCast VR Marketing System, which includes a set of tools and an SDK to simplify and streamline the problems above so that anyone can set up their own mixed reality capture system very quickly and easily. MixCast supports both greenscreen and greenscreen free mixed reality capture utilizing Intel® RealSense™ Depth Cameras. Their system allows for real time viewing of a VR experience in mixed reality. Use cases for MixCast include VR arcades and other location-based VR entertainment venues where groups of people can attend a VR arcade or LBE and watch what their friends are doing in VR in real time, without needing to wear a headset to do so.

Another great feature of the MixCast solution is the ability to give guests videos or photos of their VR experiences. Theme parks have created this kind of memorabilia for attendees for many years, but with MixCast, any location-based VR entertainment center can provide a virtual photo-booth experience, and provide videos and photos of someone playing or experiencing a particular game. Players can share these digital pictures and videos with their friends and family, and help to increase understanding and enthusiasm for VR experiences.

From a technical standpoint, the MixCast SDK is easy to use and set up, and contains some algorithms to improve background segmentation edges, giving a cleaner look to the outlines of the players.

Try MixCast out for free today!