On Friday, researchers from Nvidia announced Magic3D, an AI model that can generate 3D models from text descriptions. After entering a prompt like, “A blue poison frog sitting on a water lily,” Magic3D creates a 3D mesh model, complete with color texture, in about 40 minutes. . With modifications, the model can be used in video games or CGI renderings.
In its academic paper, Nvidia created Magic3D in response to DreamFusion, a text-to-3D model announced by Google researchers in September. Just as DreamFusion uses a text-to-image model to generate 2D images and edit them into volumetric NeRF (Neural radiance field) data, Magic3D uses a two-step process that takes the rough model is created in low resolution and optimized. to a higher resolution. According to the authors of the paper, the Magic3D method can produce 3D objects twice as fast as DreamFusion.
Magic3D also allows for quick editing of 3D screens. On a low-resolution 3D model with high speed, the text can be modified to change the resulting model. Also, Magic3D authors demonstrate keeping the same subject across multiple layers (a concept called interlacing) and applying a 2D image (like a cubist painting) to one 3D models.
Nvidia did not release any Magic3D code with its knowledge sheet.
The ability to generate 3D from text appears to be a natural evolution in today’s computational models, which use neural networks to synthesize new objects after intensive physical training. of data. In 2022 alone, we have seen the emergence of powerful text-to-image models such as DALL-E and Stable Diffusion and text-to-video generators from Google and Meta. Google also started the DreamFusion word-to-3D model two months ago, and since then, people have adapted similar methods to work as an open source model based on Stable Diffusion.
For Magic3D, the researchers behind it hope that anyone can create 3D models without the need for special training. Once refined, the resulting technology could speed up video game (and VR) development and may even find applications in special effects for film and television. Near the end of their paper, they wrote, “Hopefully with Magic3D, we can eliminate 3D manipulation and unlock everyone’s creativity to create 3D content.”