What if the development of a 3D gaming world was as simple as taking pictures with your phone? Nvidia researchers have recently developed an AI system that can predict a complete 3D model from any 2D image.
With the label "DIB-R," the AI takes a picture of any 2D object – such as a bird's picture – and predicts what it would look like in three dimensions. This prediction includes lighting, texture and depth.
DIB-R stands for differentiable interpolation-based rendering, ie it combines what it "sees", a 2D image and conclusions based on a 3D world understanding. This is strikingly similar to the way in which humans transform our 2D input into a mental 3D image.
According to Nvidia, this research has many implications for the field of robotics:
An autonomous robot must interact safely and efficiently With its environment, it must be able to sense and understand its environment. DIB-R could potentially improve these depth perception abilities.
With the advancement, the researchers hope to expand DIB-R to capabilities that would essentially make it a virtual reality renderer. One day, the team hopes, such a system will allow the AI to create fully immersive 3D worlds in just milliseconds with photos:
Sanja Fidler, Nvidia's director of AI and co-author in the team's newspaper, said Venture Beat's Khari Johnson:
Imagine, you can just take a photo and get a 3D model. This means that you can now view the scene where you took a picture of [from] from different angles. You may be able to look at it from different angles – you can take old photos in your photo collection and turn them into a 3D scene, and basically examine them as if you were there.
The ability to render the world from photos could lead to amazing content creation pipelines. Technologies like Google Maps could be more insistent than ever. And maybe creatives who are more familiar with photography or painting than with coding and development can leave all the heavy development to the machines.
Imagine you are making huge open-world games like Skyrim and Grand Theft Auto which was traditionally used in companies with hundreds of employees was something that a Handful of creatives and an AI could handle on their own.