Virtual Depth

My director, Tamer Eskander, and I are co-inventors of a patent regarding the Systems and Methods for Generating an Interactive 3D Environment Using Virtual Depth. This methodology uses a combination of pre-rendered video and invisible collision data aligned with a camera to provide the illusion of interacting with highly detailed scenes (like you would see in movies) in real-time on weaker devices, such as phones and tablets. The abstract is as follows:

Described herein are apparatuses, systems and methods for generating an interactive three-dimensional ("3D") environment using virtual depth. A method comprises receiving a pre-rendered media file comprising a plurality of frames, receiving depth data related to the media file, wherein the depth data corresponds to each of the plurality of frames, creating an invisible three-dimensional ("3D") framework of a first frame of the media file based on the corresponding depth data, and rendering a new first frame in real time to include the pre-rendered first frame, one or more virtual visible 3D objects and the invisible 3D framework.

You may read the full text of US Patent US20180047207A1 on Google Patents. In terms less technical than the patent, Virtual Depth recreates the things you see in a video as invisible objects that can use much less memory and be processed more quickly. Then an application uses that information to allow people to interact within the video in real-time. This technology transforms the ability to use highly-detailed interactive video on weaker devices, like cell phones and tablets. The following silent video by my director (probably private due to YouTube's changes regarding unlisted videos) shows an example:

To provide another example beyond this video, imagine you're watching a high-definition video on your cell phone or tablet. At one point the video stops and prompts you to choose between a soccer ball, basketball, or tennis ball. You are then able to bounce that ball off the tables and behind the couches in the video. The ability to draw everything on the screen realistically, and in real-time, is usually impossible for smaller devices to do. However, with Virtual Depth, your phone only needs to worry about how and where to draw the ball, as the video has already drawn everything else.

This patent family has resulted in 4 U.S. patents, 2 Canadian patents, and 1 European patent (all based on the original application). There is still one European application pending.