New AI method turns 2D images into 3D CAD models with 80% greater efficiency

Full text

New research explains a method that helps AI make 3D CAD programs from images, which would help product designers and engineers more quickly and easily create 3D prototypes. Naturally, this also sparks deep-rooted dread over it helping robots to build themselves, relegating humanity to the trash. But setting aside my noir nightmares, the new method looks like it could be of significant benefit to some 3D designers and engineers.

According to the paper [PDF] , the method the researchers have come up with "reduces dependence on inference compute" and performs much more efficiently than the usual 'supervised fine-tuning' (SFT) approach—up to 80% more efficient, in fact.

In the abstract for the paper, the researchers explain that making CAD programs from images "requires alignment between visual geometry and symbolic program representations" which current training methods can't easily or cheaply handle. The problem, they claim, is in "the scarcity of diverse training examples."

"Existing finetuning approaches rely on either limited supervised datasets or expensive post-training pipelines, resulting in brittle systems that restrict progress in generative CAD design. We argue that the primary bottleneck lies not in model or algorithmic capacity, but in the scarcity of diverse training examples that align visual geometry with program syntax."

The solution seems to be to let the model take its mistakes and use those as training data. Lead author and Red Hat researcher Giorgio Giannone explains: "We want engineers to be able to point our framework at an underperforming CAD model, set a compute budget, and let the system take over—turning the model’s own mistakes into better training data."

Senior co-author prof. Faez Ahmed expands: "What excites me about this work is that it gives many image-to-CAD-code models a way to improve themselves, learning from their own errors rather than waiting for more human-made data—and that brings trustworthy AI design tools much closer to everyday engineering."

(Image credit: San Francisco Chronicle/Hearst Newspapers / Contributor via Getty Images) The solution described in the research, which was funded in part by the MIT-IBM Computing Research Lab, is 'Geometric Inference Feedback Tuning' (GIFT). This asks the AI model to solve CAD generation multiple times and then augment any almost-correct solutions to become correct ones. Part of the benefit of this way of doing it is the model essentially generates its own training data automatically.

The result is training that can be up to 80% more efficient than the traditional method, as explained in the paper: "GIFT matches the peak performance of the SFT model (achieved via extensive rejection sampling) while reducing the inference compute requirement by approximately 80%."

Peak performance that's up to 80% more efficient, and all done without the need for human input... lovely stuff, but I just can't help returning to the image of those robots turning to us under a smoggy, industrial twilight and saying, ' We won't be needing you anymore.'