InstructMesh, a new software tool, allows users to refine and prepare AI-generated 3D models for physical fabrication. The tool, developed by researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) in collaboration with Google and Northeastern University, aims to bridge the gap between visually plausible AI designs and functional, printable objects. Generative AI models often produce 3D designs that appear correct visually but contain structural flaws that would render them non-functional when fabricated.
The InstructMesh system integrates Microsoft's TRELLIS 3D generator with the large language model GPT-4. This combination allows users to generate a 3D model from a text prompt and then interactively refine specific areas of the design. Users can highlight a region of the model and describe the desired change using natural language, or they can use sliders for precise geometric adjustments such as altering thickness or extruding a part. This approach enables users to apply corrections without requiring extensive 3D modeling expertise.
A key motivation for InstructMesh stemmed from the observation that many AI-generated 3D models suffer from fabrication-related defects. In a study, the research team had TRELLIS recreate popular 3D models from Thingiverse, a platform for 3D printable designs. Nearly 80 percent of the generated models exhibited structural flaws. When novices used InstructMesh to identify and repair these issues, they succeeded approximately 90 percent of the time, as verified by an expert. This indicates the tool's effectiveness in empowering users without specialized skills.
The research team demonstrated InstructMesh's capabilities by fabricating various household items and functional objects. Examples include a mug with a dragon's tail forming the handle, butterfly-wing glasses, a shell-shaped whistle, and an octopus-style dispenser designed to pour liquid from each tentacle. Beyond creative applications, InstructMesh was also used to design a denim-textured knee brace and a shrimp-like enclosure for a bristle-bot, a small robot with an internal motor.
Faraz Faruqi, an MIT PhD graduate and lead author on the paper, stated that the goal was to combine the strengths of 3D generators and large language models to create objects that people genuinely desire. Faruqi, now at Google, suggested that future iterations of InstructMesh might incorporate physics simulations to predict how designs react to specific uses, such as a bowl breaking if dropped, and to recommend suitable materials. The software may also integrate with TRELLIS.2 for more detailed feature refinement in 3D models.
The researchers will present their work at the ACM Symposium on User Interface Software and Technology in November. The project received support from Google and the MIT-HPI Collaborative Research Program.