Nvidia AI produces 3D models from 2D videos, 3D printing applications forthcoming?

US based GPU manufacturer Nvidia has published a research paper outlining a new artificial intelligence (AI) model capable of converting 2D video clips into detailed 3D models.

The research found that current generative AI models struggle to capture fine details of objects, such as repetitive texture patterns, homogenous colors, and strong color variations. Nvidia's new AI technology, called Neuralangelo, is said to surpass these problems. Neuralangelo offers high fidelity models and the ability to accurately translate the textures of complex materials from 2D videos.

How might Nvidia AI as a 3D printing tool develop?

Nvidia highlights the potential benefits for creative professionals in particular, who can quickly and easily create virtual objects using footage captured from a smartphone. Art, virtual reality, video game development, robotics, and the production of industrial digital twins have been outlined as potential applications of Neuralangelo.

The company has already demonstrated the wide ranging capabilities of their new AI technology, recreating Michelangelo's David sculpture and a flatbed truck. Neuralangelo can also reconstruct large scale environments. Nvidia has successfully created a 3D model of the park at their Bay Area campus using birdseye video footage captured by a drone.

"The 3D reconstruction capabilities Neuralangelo offers will be a high benefit to creators, helping them recreate the real world in the digital world," commented Ming-Yu Liu, Senior Director of Research at Nvidia, and co-author of the research paper. "This tool will eventually enable developers to import detailed objects – whether small statues or massive buildings – into virtual environments for video games or industrial digital twins."

While this new AI tool is yet to be combined with 3D printing applications, an Nvidia spokesperson told 3D Printing Industry this could be a future direction. Currently, Neuralangelo's models cannot be 3D printed.

It's easy to imagine the potential for additive manufacturing integration with Neuralangelo.

Whether through exporting the AI generated mesh as a printable file in the future, or as part of a digital twin. Presently, digital twins are leveraged for the design and prototyping stages of product development, and further into the product life cycle.

Converting 2D videos into 3D models with AI

Neuralangelo employs Instant Neural Graphics Primitives (NGP) in combination with signal distance functions (SDF) to achieve high quality video conversion. NGP provides hybrid 3D grid structures with a multi resolution hash encoding. This hybrid representation significantly increases the representation power of neural fields, and can capture more fine-grained details.

Effectively, Neuralangelo takes a 2D video of an object or environment, and selects several frames that highlight different viewpoints. Once the camera position of each frame is determined, the AI creates a rough 3D model. This rough rendering is then optimized, sharpening up the details and producing a finished product with high surface quality and accuracy.

The researchers highlight two factors that are key to this approach. Firstly, using numerical gradients to compute high-order derivatives. This is said to be critical for stabilizing the optimization process. Secondly, a progressive optimization schedule is vital for recovering structures at different levels of detail. This allows for comprehensive and accurate 3D model reconstruction.

Nvidia's Neuralangelo project is set to be presented at this year's Computer Vision and Pattern Recognition (CVPR) conference in Vancouver between June 18-22. Neuralangelo will be presented alongside nearly 30 other projects, with topics ranging from pose estimation, 3D reconstruction and video generation.

AI and additive manufacturing

Artificial intelligence is one of the main buzzwords within 3D printing at the minute. This year's 3D Printing Industry Executive Survey highlighted automated AI 3D model generation as part of the future of 3D printing.

Is that future coming sooner than expected? Nvidia announced an AI 3D model generating tool called Magic3D. From a text prompt Magic3D produces 3D objects within 40 minutes, double the speed of DreamFusion. Also, Materialise released its metal 3D printing Process Control, and the Build Process Software Development Kit. AI in Materialise Process Control automates the layer data analysis, saving time and money.

In a 3D Printing Industry interview with Paul Powers, founder of Thangs, he spoke about the future role of AI and 3D design.

Subscribe to the 3D Printing Industry newsletter to ensure you keep up with the latest 3D printing news. You can also follow us on Twitter, like our Facebook page, and subscribe to the 3D Printing Industry Youtube channel to access more exclusive content.

Are you interested in working in the additive manufacturing industry? Visit 3D Printing Jobs to view a selection of available roles and kickstart your career. Featured image shows a Michaelangelo statue made into a 3D model using Neuralangelo. Image via NVIDIA.

Alex is a Technology Journalist at 3D Printing Industry who enjoys researching and writing articles covering a wide variety of topics. Possessing a BA in military history and an MA in History of War, he has a keen interest in additive manufacturing applications within the defense and aerospace industries.