Google engineers have developed a technology that gives you the opportunity to demonstrate to the viewer the most realistic surround video. Thanks to the polymer hemisphere on which the action Cam, a person can see to consider what is happening in the video from different angles and directions. Article about this development will be presented at SIGGRAPH 2020, scheduled for mid-July.
A person sees the world is big due to the fact that he has two eyes and visual cortex of the brain receives the image from two different points in space. And due to the activated movements of the head-motion parallax (projection changes on the retina when you move in space), and it gives us the opportunity to assess the volume and offset of the objects relative to each other.
The stereo cameras and the virtual reality helmets reproduces binocular vision because each eye receives frames from different angles. However, motion parallax to reproduce this approach is impossible, because when the camera was at a specific point, and change it after it is already impossible.
Google engineers are working on a solution to this problem is already not the first year. Thanks to the combination of hardware and software methods, they managed to create technology that gives you the opportunity to consider what is happening in the video as if from different angles. Footage from the 46 cameras positioned on the hemispherical surface, is transmitted to the convolutional neural network, which, in turn, divides these frames into multiple layers, depending on the distance to a particular object in the field shooting.
A fragment of one of the videos
Each video is divided into 160 layers with a resolution of 1800 1350 pixels, and then these layers are optimized: every eight layers are combined with the creation of polygon meshes, and the image as a texture overlaid on top. The obtained textures are combined into texture Atlas, the resolution of which is at 3240 5760 pixels. Thus, the initial data stream, where each frame corresponds to multiple layers, transforms into two separate flow: the first contains images that can efficiently compress, and the second polygon mesh.
The scheme of receiving video using a hemispherical camera and subsequent image processing
Video about developing
Using this approach, developers have created several videos in which you can change the angle movements of the cursor. The bitrate of the stream (the number of bits used for the processing and transmission of data per unit of time) is in the range from 150 to 300 megabits per second. This gives you the opportunity to apply the technology to stream realistic surround video virtual reality with Gigabit Internet.