3D Texture Synthesis Using Graph Neural Cellular Automata

Abstract: In recent years, texture synthesis has been a heated topic in computer graphics, and the development of advanced algorithms for generating high-quality 3D textures is an area of active research. A recently proposed model, Neural Cellular Automata, can synthesize realistic 2D texture images or videos. However, due to the complexity and non-differentiable nature of 3D rendering and the lack of definition of the neighborhood on 3D mesh objects, no one has extended the 2D Neural Cellular Automata to the 3D scenario. In this master’s thesis, we propose a novel method for modeling the neighborhood relationship on 3D mesh objects, drawing inspiration from a graph variant of the Neural Cellular Automata. We also design an end-to-end 3D texture synthesis pipeline, leveraging a differentiable renderer to enable the Graph Neural Cellular Automata to learn to synthesize desired 3D textures. Our method allows users to either give the text description of the target textures or present the target texture images as the objectives. We evaluate the effectiveness of our proposed method both qualitatively and quantitatively, comparing it with the state-of-the-art method to demonstrate that it achieves comparable or better results. Furthermore, we explore the homology between the graph variant of Neural Cellular Automata and the 2D model, examining whether our proposed model preserves critical properties of the 2D model such as zero-shot generalization and self-regeneration. Finally, we analyze the limitations and potential drawbacks of our proposed method and suggest directions for future research. In summary, this thesis proposes a novel approach to synthesizing high-quality 3D textures using the Graph Neural Cellular Automata model and a differentiable renderer. Our work provides a foundation for future research in this area, and we believe that our findings will contribute to the development of advanced algorithms for 3D texture synthesis.