# LongBench v2 / 66ebee0a5a08c7b9b35e1d05

task_id: 9c5f8d95-e6b6-5d14-890f-89b3d02a83c2
task_key: train--66ebee0a5a08c7b9b35e1d05
task_revision_id: 3

{"choice_A":"Meta-ControlNet introduces alignment-weighted gradients where the similarity between the 3D reference and the concept image (measured by cosine similarity) is used to dynamically scale the gradients in backpropagation. If the reference and image are misaligned, it reduces the gradient contribution from the reference, preventing the model from fitting erroneous geometrical details. This modulation happens across almost all noise levels to guarantee that both global and local features are learned without overfitting to poor references.","choice_B":"Meta-ControlNet applies time-dependent gradient scaling, where at higher timesteps (when the noise level is higher), the reference model is given more influence on gradient updates through increased weight on its canonical coordinate maps (CCMs). This forces the model to hallucinate missing parts of the 3D object when the reference is not closely aligned with the concept image. As the noise level declines, the model shifts to rely more on the image, prioritizing the image’s geometric integrity during backpropagation at later stages.","choice_C":"Meta-ControlNet incorporates an auxiliary loss term based on the L2 distance between the reference and concept image features. This term is minimized during backpropagation to encourage the model to forcefully align the concept image and reference model even when there is a mismatch. The result is stronger gradients for references that are dissimilar, which improves the ability of the model to learn generalizable shape priors from misaligned references.","choice_D":"Meta-ControlNet modulates multi-scale feature alignment using a learned weighting matrix that dynamically scales the gradients according to both the noise level and the feature similarity between the reference and the concept image. At high noise levels, the matrix suppresses the gradients from the reference model to avoid distorting the overall geometry, while at low noise levels, it increases the gradient influence from the reference to refine local details. This allows for controlled generation based on the level of alignment across different noise stages of diffusion.","context":"PHIDIAS: A GENERATIVE MODEL FOR CREATING 3D\nCONTENT FROM TEXT, IMAGE, AND 3D CONDITIONS\nWITH REFERENCE-AUGMENTED DIFFUSION\n\nABSTRACT\nIn 3D modeling, designers often use an existing 3D model as a reference to create\nnew ones. This practice has inspired the development of Phidias, a novel gen-\nerative model that uses diffusion for reference-augmented 3D generation. Given\nan image, our method leverages a retrieved or user-provided 3D reference model\nto guide the generation process, thereby enhancing the generation quality, gen-\neralization ability, and controllability. Our model integrates three key compo-\nnents: 1) meta-ControlNet that dynamically modulates the conditioning strength,\n2) dynamic reference routing that mitigates misalignment between the input image\nand 3D reference, and 3) self-reference augmentations that enable self-supervised\ntraining with a progressive curriculum. Collectively, these designs result in signif-\nicant generative improvements over existing methods. Phidias establishes a uni-\nfied framework for 3D generation using text, image, and 3D conditions, offering\nversatile applications. Demo videos are at: https://RAG-3D.github.io/.\nFigure 1: The proposed model, Phidias, can produce high-quality 3D assets given 3D references,\nwhich can be obtained via retrieval (top two rows) or specified by users (bottom row). It supports\n3D generation from a single image, a text prompt, or an existing 3D model.\n1\nINTRODUCTION\nThe goal of 3D generative models is to empower artists and even beginners to effortlessly convert\ntheir design concepts into 3D models. Consider the input image in Fig. 1. A skilled craftsman can,\nthrough a blend of skills and creativity, convert a 2D concept image into an exquisite 3D model. This\ncreative process can originate from artists’ pure imagination or, more commonly, through examining\n†Intern at Shanghai AI Lab. ∗Equal Contribution.\n1\narXiv:2409.11406v1  [cs.CV]  17 Sep 2024\n\n\none or more existing 3D models as a source of inspiration (Bob, 2022; Carvajal, 2023). Artists often\nrefer to these pre-existing 3D models to improve the modeling quality. The question then arises:\ncould we develop a reference-based 3D generative model that can replicate this capability?\nOver the years, a plethora of works (Wang et al., 2023; Liu et al., 2023b; Hong et al., 2023; Ben-\nsadoun et al., 2024) steadily expanded the frontiers of 3D generative models. These methods, while\nyielding stunning performance, still face several challenges. 1) Generation quality. A single im-\nage cannot furnish sufficient information for reconstructing a full 3D model, due to the ambiguity\nof this ill-posed task. This necessitates the generative model to “hallucinate” the unseen parts in a\ndata-driven manner. However, this hallucination can lead to view inconsistency and imprecise ge-\nometries that appear abrupt and unrealistic. 2) Generalization ability. These models often struggle\nwith out-of-domain cases, such as atypical input views or objects, constrained by the data coverage\nof existing 3D datasets (Deitke et al., 2023). Also, the growing variety and quantity of object cate-\ngories exacerbate the difficulty for generative models to learn implicit shape priors, with a limited\nmodel capacity v.s. an infinitely diverse array of objects. 3) Controllability. Due to the ambiguity,\none input image can produce several plausible 3D models, each differing in shape, geometric style,\nand local patterns. Existing methods are constrained by limited diversity and controllability, which\nhinders the ability to predictably generate the desired 3D models.\nTo address these challenges, we propose to take 3D models as additional inputs to guide the gener-\nation, inspired by the success in retrieval augmented generation (RAG) for language (Lewis et al.,\n2020) and image (Sheynin et al., 2022). Given an input image and a reference 3D model, we present\nPhidias, a novel reference-augmented diffusion model that unifies 3D generation from text, image,\nand 3D conditions. As shown in Fig. 1, the reference 3D model would help 1) improve quality\nby alleviating ambiguity with richer information for unseen views, 2) enhance generalization ca-\npacity by serving as a shape template or an external memory for generative models, and 3) provide\ncontrollability by indicating desired shape patterns and geometric styles.\nOur method proposed a reference-augmented multi-view diffusion model, followed by sparse-view\n3D reconstruction. The goal is to produce 3D models faithful to the concept image with improved\nquality by incorporating relevant information from the 3D reference. However, it is non-trivial to\nlearn such a generative model due to the Misalignment Dilemma, where the discrepancy between the\nconcept image and the 3D reference can lead to conflicts in the generation process. This requires our\nmodel to utilize the misaligned 3D reference adaptively. To tackle this challenge, Phidias leverages\nthree key designs outlined below.\nThe first is meta-ControlNet. Consider 3D reference as conditions for diffusion models. Unlike\nprevious image-to-image translation works (Zhang et al., 2023; Wang et al., 2022) that demand the\ngenerated images to closely follow the conditions, we treat reference model as auxiliary guidance to\nprovide additional information. The generated multi-view images are expected to be consistent with\nthe concept image, without requiring precise alignment with the reference model. To this end, we\nbuild our method on ControlNet and propose a meta-control network that dynamically modulates\nconditioning strength when it conflicts with the concept image, based on their similarity.\nThe second design is dynamic reference routing for further alleviating the misalignment. Rather\nthan using the same 3D reference for the full diffusion process, we adjust its resolution across\ndenoise timesteps. This follows the dynamics of the reverse diffusion process (Balaji et al., 2022),\nwhich generates coarse structure in high-noised timesteps and details in low-noised timesteps. Thus,\nwe can alleviate the generation conflicts by starting with a coarse 3D reference and progressively\nincreasing its resolution as the reverse diffusion process goes on.\nThe final key design is self-reference augmentations. It is not feasible to gather large sets of 3D\nmodels and their matching references. A practical solution is to use the 3D model itself as its own\nreference (i.e., self-reference) for self-supervised learning. The trained model, however, does not\nwork well when the 3D reference does not align with the target image. To avoid overfitting to a\ntrivial solution, we apply a variety of augmentations to 3D models that simulate this misalignment.\nFurthermore, we introduce a progressive augmentation approach that leverages curriculum learning\nfor diffusion models to effectively utilize references that vary in similarity.\nTaken together, the above ingredients work in concert to enable Phidias to achieve stunning perfor-\nmance in 3D generation. Several application scenarios are thus supported: 1) Retrieval-augmented\n2\n\n\nFigure 2: Overview of the Phidias model. It generates a 3D model in two stages: (1) reference-\naugmented multi-view generation and (2) sparse-view 3D reconstruction.\nimage-to-3D generation, 2) Retrieval-augmented text-to-3D generation, 3) Theme-aware 3D-to-3D\ngeneration, 4) Interactive 3D generation with coarse guidance, and 5) High-fidelity 3D completion.\nWe summarize our contributions as follows: 1) We propose the first reference-based 3D-aware diffu-\nsion model. 2) We design our model with three key component designs to enhance the performance.\n3) Our model serves as a unified framework for 3D generation, which provides a variety of appli-\ncations with text, image, and 3D inputs. 4) Extensive experiments show our method outperforms\nexisting approaches qualitatively and quantitatively.\n2\nRELATED WORKS\nImage to 3D. Pioneering works (Melas-Kyriazi et al., 2023; Tang et al., 2023; Chen et al., 2024b)\nperform 3D synthesis by distilling image diffusion priors (Poole et al., 2023), but are time-\nconsuming. Recent advancements have leveraged feed-forward models with 3D datasets. Some\nworks use diffusion models to generate points (Nichol et al., 2022), neural radiance fields (Wang\net al., 2023; Jun & Nichol, 2023; Gupta et al., 2023; Hong et al., 2024), SDF (Cheng et al., 2023;\nZhang et al., 2024b), and gaussian splatting (Zhang et al., 2024a). Another line of works uses trans-\nformers for auto-regressive generation (Siddiqui et al., 2023; Chen et al., 2024a) or sparse-view\nreconstruction (Hong et al., 2023; Tang et al., 2024; Zou et al., 2023; Wang et al., 2024a; Xu et al.,\n2024), which often rely on multi-view diffusion for better performance.\nMulti-View Diffusion Models. Multi-view models reduce the complexities of 3D synthesis to con-\nsistent 2D synthesis. Seminal works (Liu et al., 2023b) have shown novel view synthesis capabilities\nwith pre-trained image diffusion models (Rombach et al., 2022). Later, a plethora of works explored\nmulti-view diffusion models with better consistency (Shi et al., 2023a; Wang & Shi, 2023; Shi et al.,\n2023b; Long et al., 2023; Liu et al., 2023a) by introducing cross-view communication. More recent\nworks (Voleti et al., 2024; Chen et al., 2024c; You et al., 2024; Han et al., 2024) leverage video pri-\nors for multi-view generation by injecting cameras into video diffusion models. However, they still\nstruggle with generalized and controllable generation due to the ill-posed nature of this problem.\nReference-Augmented Generation. Retrieval-augmented generation (RAG) emerges to enhance\nthe generation of both language (Lewis et al., 2020) and image (Sheynin et al., 2022; Blattmann\net al., 2022) by incorporating relevant external information during the generation process. Under\nthe context of 3D generation, the concept of reference-based generation is also widely applied.\nSome works (Chaudhuri et al., 2011; Kim et al., 2013; Schor et al., 2019) probe into the database for\ncompatible parts and assemble them into 3D shapes. Some works refer to a 3D exemplar model (Wu\n& Zheng, 2022; Wang et al., 2024b) to produce customized 3D assets. Despite success in specific\ncontexts, they are time-consuming with per-case optimization. In contrast, our method focuses on\nlearning a generalized feed-forward model that applies to reference-augmented 3D generation.\n3\nAPPROACH\nGiven one concept image, we aim at leveraging an additional 3D reference model to alleviate 3D\ninconsistency issues and geometric ambiguity that exist in 3D generation. The 3D reference model\ncan be either provided by the user or retrieved from a large 3D database for different applications.\n3\n\n\nLow Noise Levels\n3D Reference \nBase ControlNet\nMulti-View CCM Image\n…\nMeta-Controller\nConcept\nImage\nZero\nConvs\nZero\nConvs\nAdaptive Control Signal\nMulti-Scale Alignment Features\nZero Convs\n3D Reference\nFront-View \nCCM\nEncoder\nEncoder\n(a) Meta-ControlNet\n(b) Dynamic Reference Routing\n…\n…\n…\n…\nMiddle Noise Levels\nHigh Noise Levels\n…\nHigh Res. CCM\nMiddle Res. CCM\nLow Res. CCM\n…\n…\n…\n…\n…\n…\n…\n𝑡!\n𝑡\"\n𝑡#\nFigure 3: Architectural designs for meta-ControlNet (a) and dynamic reference routing (b).\nThe overall pipeline of Phidias is shown in Fig. 2, which involves two stages: reference-augmented\nmulti-view generation and sparse-view 3D reconstruction.\n3.1\nREFERENCE-AUGMENTED MULTI-VIEW DIFFUSION\nMulti-view diffusion models incorporate camera conditions into well-trained image diffusion mod-\nels for novel-view synthesis with supervised fine-tuning. We aim to weave additional 3D references\ninto these multi-view models for better generation quality, generalization ability, and controllability.\nOur approach can be built on arbitrary multi-view diffusion models, enabling reference-augmented\n3D content creation from text, image, and 3D conditions. Specifically, we initialize our model with\nZero123++ (Shi et al., 2023a), which simply tiles multi-view images for efficient generation condi-\ntioned on one input image cimage.\nTo integrate 3D reference models cref into the diffusion process, we transform them into multi-view\ncanonical coordinate maps (CCM) to condition the diffusion model. The choice of CCMs as the 3D\nrepresentation is based on two reasons: 1) Multi-view images serve as more efficient and compatible\ninputs for diffusion models than meshes or voxels, as they have embedded camera viewing angles\nthat correspond with the output images. 2) Reference models often share similar shapes with the\nconcept image but vary significantly in texture details. By focusing on the geometry while omitting\nthe texture, CCMs conditions can reduce generation conflicts arising from texture discrepancies. We\nadd a conditioner branch to incorporate reference CCMs into the base multi-view diffusion model.\nThe objective for training our diffusion model ϵθ can be then formulated as:\nL = Et,ϵ∼N (0,1)\n\u0002\n∥ϵ −ϵθ (xt, t, cimage, cref) ∥2\u0003\n(1)\nTo leverage the powerful pertaining capability, only the additional conditioner for reference CCMs\nis trainable while the base multi-view diffusion is frozen. However, a challenge in our task is that the\n3D reference may not strictly align with the concept image or, more commonly, vary in most local\nparts. We found naive conditioner designs such as ControlNet (Zhang et al., 2023) tend to produce\nundesirable artifacts, as they were originally designed for image-to-image translation where the gen-\nerated images strictly align with the condition images. To mitigate this problem, we introduce three\nkey designs for our reference-augmented diffusion model: (1) Meta-ControlNet for adaptive control\nof the conditioning strength (Sec. 3.2); (2) Dynamic Reference Routing for dynamic adjustment of\nthe 3D reference (Sec. 3.3); (3) Self-Reference Augmentation for self-supervised training (Sec. 3.4).\n3.2\nMETA-CONTROLNET.\nControlNet is designed to add additional controls to pre-trained diffusion models for image-to-image\ntranslation. The conditions are derived from the ground-truth images for self-supervised learning,\nand thus the generated images are expected to follow the conditions. However, in our settings, the\nconditions are from the reference model, which often misaligns with the target 3D models we want\nto generate. The vanilla ControlNet fails to handle such cases. This necessitates further architecture\nadvancement to accordingly adjust conditioning strength when the reference conflicts with the con-\ncept image. To this end, we propose meta-ControlNet, as shown in Fig. 3 (a). Meta-ControlNet is\ncomprised of two collaborative subnets, a base ControlNet and an additional meta-controller.\n4\n\n\nBase ControlNet is comprised of an image encoder, a trainable copy of down-sampling blocks and\nmiddle blocks of the base multi-view diffusion, denoted as Fbase\nΘ\n(·), and a series of 1 × 1 zero\nconvolution layers (Zero Convs) Zbase\nΘ\n(·). It takes reference CCM maps cref as input to produce the\ncontrol signal. To deal with misaligned 3D reference, we introduce an additional meta-controller to\nmodulate the conditioning strength according to different similarity levels.\nMeta-controller shares a similar architecture but has different parameters Θ′. It works as a knob that\ndynamically modulates base ControlNet to generate adaptive control signals. Meta-controller takes a\npair cpair of the concept image and the front-view reference CCM as input to produce meta-control\nsignals based on their similarities. The meta-control signals are injected into diffusion models in\ntwo ways. On the one hand, meta-controller produces multi-scale alignment features ymeta1 =\nZmeta1\nΘ′\n(Fmeta\nΘ′\n(zpair)) to be injected into base ControlNet. These features are applied to the down-\nsampling blocks of base ControlNet (Eq. 2) at each scale to guide the encoding of reference and help\nproduce base-signals as:\nybase = Zbase\nΘ\nFbase\nΘ\n(ymeta1, zref)\n\u0001\n,\n(2)\nwhere zref and zpair are the feature maps of cref and cpair via the trainable encoders in Fig. 3 (a).\nOn the other hand, meta-controller produces meta-signals ymeta2 = Zmeta2\nΘ′\n(Fmeta\nΘ′\n(zpair)) to\nbe injected to the pretrained multi-view diffusion models. These features are added up to base-\nsignal ybase to directly apply for the pretrained diffusion models. Totally, the final outputs of meta-\nControlNet are adaptive control signals yadaptive based on the similarity between the concept image\nand the 3D reference, as:\nyadaptive = ybase + ymeta2.\n(3)\n3.3\nDYNAMIC REFERENCE ROUTING\nReference models typically align roughly with the concept image in terms of coarse shape, but\ndiverge significantly in local details. This misalignment can cause confusion and conflicts, as the\ngeneration process relies on both the image and reference model. To address this issue, we propose\na dynamic reference routing strategy that adjusts the reference resolution across denoise timesteps,\nas shown in Fig. 3 (b). As widely observed during the reverse diffusion process, the coarse structure\nof a target image is determined in high-noised timesteps and fine details emerge later as the timestep\ngoes on. This motivates us to start with low-resolution reference CCMs at high noise levels th. By\nlowering the resolution, reference models provide fewer details but exhibit smaller misalignment\nwith the concept image. This enables reference models to assist in generating the global structure\nof 3D objects without significant conflicts. We then gradually increase the resolution of reference\nCCMs as the reverse diffusion process goes into middle noise levels tm and low noise levels tl to\nhelp refine local structures, e.g., progressively generating a curly tail from a straight one (Fig. 3 (b)).\nThis design choice would ensure effective usage of both concept image and 3D reference during the\nmulti-view image generation process while avoiding degraded generation caused by misalignment.\n3.4\nSELF-REFERENCE AUGMENTATION\nA good reference model should resemble the target 3D model (with varied details) to provide addi-\ntional geometric cues, but it is impractical to collect sufficient target-reference pairs for training. An\nintuitive solution is to retrieve a similar model from a large 3D database as the training reference.\nHowever, due to the limited variety in current databases, finding a perfect match is challenging. The\nretrieved reference can vary greatly in orientation, size and semantics. While this is a common situ-\nation in inference scenarios, where a very similar reference is often unavailable, we found training\nwith these challenging pairs fails to effectively use the 3D reference. We conjecture that the learning\nprocess struggles due to the significant differences between the reference and target 3D, leading the\ndiffusion model to disregard the references. To avoid the ‘idleness’ of reference, we developed a\nself-reference scheme that uses the target model as its own reference by applying various augmen-\ntations to mimic misalignment (refer to Appendix A.4). This approach ensures that the reference\nmodels are somewhat aligned with the target and more compatible, alleviating the learning difficulty.\nWe further design a curriculum training strategy, which begins with minimal augmentations (very\nsimilar references) to force the diffusion model to rely on the reference for enhancement. Over time,\nwe gradually increase augmentation strength and incorporate retrieved references, challenging the\n5\n\n\nInput Image\nRetrieved\n3D Reference 1\nGenerated Model 1\nRetrieved\n3D Reference 2\nGenerated Model 2\nFigure 4: Diverse retrieval-augmented image-to-3D results. Phidias can generate diverse 3D models\nwith different references for a single input image.\ndiffusion model to learn from references that do not closely match the target. Once trained, our\nmodel performs well with a variety of references, even those retrieved ones that are not very similar.\n3.5\nSPARSE-VIEW 3D RECONSTRUCTION\nWith multi-view images generated in the first stage, we can obtain final 3D models via sparse-\nview 3D reconstruction. This step can be built upon arbitrary sparse-view reconstruction models.\nSpecifically, we finetune LGM (Tang et al., 2024) by expanding the number of input views from 4\nto 6 and the resolution of each view from 256 × 256 to 320 × 320 so that the trained reconstruction\nmodel aligns with the multi-view images generated in our first stage.\n4\nEXPERIMENTS\nIn this section, we evaluate our method on image-to-3D generation, a significant area in 3D gen-\neration research. For each image, we retrieve a 3D reference model from a 3D database based on\nsimilarity (Zhou et al., 2024). The database used is a subset of Objaverse, containing 40K models.\nWe anticipate that performance could be further enhanced with a larger database in the future. For\nthe rest of this section, we compare Phidias with state-of-the-art methods and conduct ablation anal-\nysis. More results and implementation details can be found in Appendix. Results on text-to-3D and\n3D-to-3D generation can be found in Sec. 5.\n4.1\nCOMPARISONS WITH STATE-OF-THE-ART METHODS\nWe compare Phidias with five image-to-3D baselines: CRM (Wang et al., 2024a), LGM (Tang et al.,\n2024), InstantMesh (Xu et al., 2024), SV3D (Voleti et al., 2024), and OpenLRM (He & Wang, 2023).\nQualitative Results. For visual diversity (Fig. 4), given the same concept image, Phidias can gener-\nate diverse 3D assets that are both faithful to the concept image and conforming to a specific retrieved\n6\n\n\nOurs\nInput\nImage + 3D\nCRM\nLGM\nInstantMesh\nSV3D\nOpenLRM\nFigure 5: Qualitative comparisons on image-to-3D generation.\nTable 1: Quantitative comparison with baselines on image-to-3D synthesis.\nMethod\nPSNR ↑\nSSIM ↑\nLPIPS ↓\nCLIP-P ↑\nCLIP-I ↑\nCD ↓\nF-Score ↑\nOpenLRM\n16.15\n0.843\n0.194\n0.866\n0.847\n0.0446\n0.805\nLGM\n14.80\n0.807\n0.219\n0.869\n0.871\n0.0398\n0.831\nCRM\n16.35\n0.841\n0.182\n0.855\n0.843\n0.0443\n0.796\nSV3D\n16.24\n0.838\n0.203\n0.879\n0.866\n-\n-\nInstantMesh\n14.63\n0.796\n0.235\n0.882\n0.880\n0.0450\n0.788\nOurs (GT Ref.)\n20.37\n0.870\n0.117\n0.911\n0.885\n0.0391\n0.840\nOurs (Retrieved Ref.)\n17.02\n0.845\n0.174\n0.887\n0.885\n0.0402\n0.833\n3D reference in geometry. For visual comparisons (Fig. 5), while the baseline methods can generate\nplausible results, they suffer from geometry distortion (e.g., horse legs). Besides, none of the exist-\ning methods can benefit from the 3D reference for improved generalization ability (e.g., excavator’s\ndipper) and controllability (e.g., cat’s tail) as ours.\nQuantitative Results. Following previous works, we conduct quantitative evaluation on google\nscanned objects (GSO) (Downs et al., 2022). We remove duplicated objects with the same shape\nand randomly select 200 objects for evaluation. For visual quality, we report reconstruction met-\nrics (PSNR, SSIM and LPIPS) on 20 novel views. We also report novel views’ CLIP similarity\nwith paired GT (CLIP-P) and input image (CLIP-I). For geometry quality, we sample 50K points\nfrom mesh surface and compute Chamfer Distance (CD) and F-Score (with a threshold of 0.05). To\nalign the generated mesh and GT, we unify their coordinate systems and re-scale them into a unit\nbox. We report our results with the retrieved reference, i.e., Ours (Retrieved Ref.), and GT mesh as\nreference, i.e., Ours (GT Ref.), respectively. As shown in Tab. 1, ours, with either retrieved or GT ref-\nerence, outperforms all baselines, benefiting from the proposed retrieval-augmented method. While\nthe CD is slightly larger, we argue that our approach produces plausible 3D models given different\nreferences (Fig. 7), though they can differ from GT mesh when computing chamfer distance.\nUser Study. We further conduct a user study to evaluate human preferences among different meth-\nods. We publicly invite 30 users to complete a questionnaire for pairwise comparisons. We show the\npreference rate (i.e., the percentage of users prefer ours compared to a baseline method) in Tab. 2,\nwhich suggests that our approach significantly outperforms existing methods in the image-to-3D\ntask based on human preferences.\n7\n\n\nTable 2: User study.\nBaseline\nPref. Rate\nOpenLRM\n94.7%\nLGM\n95.8%\nCRM\n93.7%\nSV3D\n88.4%\nInstantMesh\n91.6%\nTable 3: Quantitative ablation study of the proposed components.\nMethod\nPSNR ↑\nSSIM ↑\nLPIPS ↓\nCLIP-P ↑\nCLIP-I ↑\nCD ↓\nF-Score ↑\nBase Model\n14.70\n0.804\n0.227\n0.855\n0.859\n0.0424\n0.826\n+ Meta-ControlNet\n16.35\n0.833\n0.190\n0.881\n0.878\n0.0407\n0.829\n+ Dynamic Ref. Routing\n14.76\n0.816\n0.221\n0.868\n0.861\n0.0420\n0.826\n+ Self-Ref. Augmentation\n16.57\n0.840\n0.182\n0.880\n0.883\n0.0414\n0.830\nFull Model\n17.02\n0.845\n0.174\n0.887\n0.885\n0.0402\n0.833\nBase Model + Retrieval\nInputs\n+ Meta-ControlNet\n(a) Meta-ControlNet\nBase Model\nInputs\n+ Dynamic Reference Routing\n(b) Dynamic Reference Routing\nBase Model\nInputs\n+ Self-Reference Augmentation\n(c) Self-Reference Augmentation\nFigure 6: Qualitative ablation study of the proposed components.\n4.2\nABLATION STUDY AND ANALYSIS\nAblation Studies. We conduct ablation studies across four settings: a base model employing a\nstandard ControlNet trained with self-reference, and three variants (each integrating one proposed\ncomponent into the base model). The quantitative results in Tab. 3 demonstrate clear improvements\nin both visual and geometric metrics with our proposed components.\nEffectiveness of Meta-ControlNet. To evaluate meta-ControlNet, we use both self-reference and\nretrieved reference for training, as the learning of Meta-Controller (Fig. 3 (a) top) requires reference\nmodels with varying levels of similarity. As shown in Fig. 6 (a), the base model trained with retrieved\nreference often ignores the reference, failing to follow the shape pattern (disconnected boat). This\nphenomenon stems from the considerable similarity variation among retrieved references, which\nconfuses the diffusion model. The base model thereby struggles to determine when and how to use\nthe reference as it lacks the ability to adjust to different levels of similarity. Consequently, they\noften end up with ignoring the reference models entirely. In contrast, meta-ControlNet equips the\nmodel with the capability to dynamically modulate the conditioning strength of the reference model,\nthereby effectively utilizing available references for improving or controlling the generation process.\nEffectiveness of Dynamic Reference Routing. Dynamic reference routing aims to alleviate local\nconflicts between the reference and concept images. As illustrated in Fig. 6 (b), when given a highly\nsimilar reference, the base model tends to rely heavily on it, leading to missing specific local details\nwithin the concept image, e.g., the rope on the left. By addressing these conflicts with dynamic\nrouting, the model maintains the essential details of the concept image, while still benefiting from\nthe guidance of the 3D reference.\nEffectiveness of Self-Reference Augmentation. As shown in Fig. 6 (c), without self-reference aug-\nmentation, the base model predominantly depends on the provided reference for generation. When\ngiven a significantly misaligned reference, the model tends to follow the reference’s structure, re-\nsulting in an undesired outcome. Conversely, self-reference augmentation ensures that the generated\nmodels remain faithful to the concept image, while using the reference as geometry guidance.\nAnalysis on Similarity Levels of 3D Reference. We analyze how similarity levels of 3D refer-\nences would affect the performance. For each input, we retrieve three models ranked first (top-1),\nthird (top-3), and fifth (top-5) in similarity scores, and randomly choose one model, to serve as 3D\nreferences. Quantitative results in Tab. 4 indicate that Phidias performs better with more similar\n8\n\n\nTable 4: Quantitative analysis on similarity levels of 3D reference.\nReference\nPSNR ↑\nSSIM ↑\nLPIPS ↓\nCLIP-P ↑\nCLIP-I ↑\nCD ↓\nF-Score ↑\nTop-1 Retrieval\n17.02\n0.845\n0.174\n0.887\n0.885\n0.0402\n0.833\nTop-3 Retrieval\n16.75\n0.841\n0.172\n0.887\n0.886\n0.0395\n0.830\nTop-5 Retrieval\n15.96\n0.835\n0.185\n0.886\n0.884\n0.0408\n0.819\nRandom Reference\n14.74\n0.820\n0.226\n0.884\n0.882\n0.0424\n0.810\nWithout Reference\n15.90\n0.836\n0.188\n0.886\n0.880\n0.0416\n0.814\nFigure 7: Qualitative analysis on similarity levels of 3D Reference.\nFigure 8: Phidias enables retrieval-augmented text-to-3D generation by first converting input text\ninto a concept image, and then retrieving a 3D reference based on both the text and image.\nreferences. Fig. 7 shows Phidias generates diverse plausible results with different references. All\nresults remain faithful to the input image in the front view, but show variations in shapes influenced\nby the specific reference used. Also, we found Phidias can still generate plausible results even with\na random 3D reference, indicating robustness to reference with different similarity levels.\n5\nAPPLICATIONS\nPhidias supports versatile applications beyond image-to-3D, such as text-to-3D, theme-aware 3D-\nto-3D, interactive 3D generation with coarse guidance, and high-fidelity 3D completion.\nText to 3D. Text-to-3D generation can be converted to image-conditioned generation by transform-\ning a text prompt into a concept image. However, the generated concept image can sometimes be\natypical and may lose some information compared with original text input. To enhance generative\nquality, Phidias employs retrieval-augmented text-to-3D generation, as illustrated in Fig. 8. This\ninvolves first retrieving a set of 3D references based on the concept image, and then selecting the\none that most closely matches the text description as the final reference.\nTheme-Aware 3D-to-3D Generation. This task aims to create a gallery of theme-consistent 3D\nvariations from existing 3D models. Previous work (Wang et al., 2024b) proposed an optimization-\nbased approach, which is time-consuming. Phidias supports fast generation by first generating im-\nage variations based on the input 3D model, and then transforming these variant images into 3D\nvariations with the original 3D model itself as reference. The results are shown in Fig. 9, using 3D\nmodels from Sketchfab1 and previous works as inputs.\nInteractive 3D Generation with Coarse Guidance. Interactive generation gives users more control\nover the outputs, empowering them to make quick edits and receive rapid feedback. Phidias also\nprovides this functionality, allowing users to continually adjust the geometry of generated 3D models\nusing manually created coarse 3D shapes as reference models, as shown in Fig. 10.\nHigh-Fidelity 3D Completion. Given incomplete 3D models, as shown in Fig. 11, Phidias can be\nused to restore the missing components. Specially, by generating a complete front view through\n1https://sketchfab.com/\n9\n\n\n3D Input\nSelf-Reference\nGenerated 3D Variation 1\nGenerated 3D Variation 2\nImage Variations\nFigure 9: Phidias facilitates rapid, theme-aware 3D-to-3D generation by using an existing 3D model\nas a reference to transform its image variations into corresponding 3D variations.\nInput Image\nCoarse Shape\nCoarse Shape\nGenerated 3D\nGenerated 3D\nFigure 10: Phidias enables interactive 3D generation with coarse 3D shapes as guidance.\nFigure 11: Phidias supports high-fidelity 3D completion by using the completed front views to guide\nthe missing parts restoration and the original 3D model to help preserve the origin details.\nimage inpainting and referencing to the original 3D model, Phidias can precisely predict and fill in\nthe missing parts in novel views while maintaining the integrity and details of the origin, resulting\nin a seamlessly and coherently structured 3D model.\n6\nCONCLUSION\nIn this work, we introduced Phidias, a 3D-aware diffusion model enhanced by 3D reference. By in-\ncorporating meta-ControlNet, dynamic reference routing, and self-reference augmentations, Phidias\neffectively leverages reference models with varying degrees of similarity for 3D generation. The\nproposed approach boosts the quality of 3D generation, expands its generalization capabilities, and\nimproves user control. Phidias offers a unified framework for creating high-quality 3D content from\ndiverse modalities, such as text, images, and pre-existing 3D models, enabling versatile applications.\nWe believe that Phidias will inspire further research to advance the field of 3D generation.\nACKNOWLEDGMENTS\nThis work is partially supported by the National Key R&D Program of China (2022ZD0160201)\nand Shanghai Artificial Intelligence Laboratory. This work is also in part supported by a GRF grant\nfrom the Research Grants Council of Hong Kong (Ref. No.: 11205620).\n10\n\n\nREFERENCES\nYogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten\nKreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with\nan ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.\nRaphael Bensadoun, Tom Monnier, Yanir Kleiman, Filippos Kokkinos, Yawar Siddiqui, Mahendra\nKariya, Omri Harosh, Roman Shapovalov, Benjamin Graham, Emilien Garreau, et al. Meta 3d\ngen. arXiv preprint arXiv:2407.02599, 2024.\nAndreas Blattmann, Robin Rombach, Kaan Oktay, Jonas M¨\nuller, and Bj¨\norn Ommer. Retrieval-\naugmented diffusion models. Advances in Neural Information Processing Systems, 35:15309–\n15324, 2022.\nBob. 3D modeling 101: Comprehensive beginners guide, 2022. URL https://wow-how.com/\narticles/3d-modeling-101-comprehensive-beginners-guide.\nCarlos\nCarvajal.\nThe\nimportance\nof\nreferences\nin\n3d\nprojects,\n2023.\nURL\nhttps://www.linkedin.com/pulse/\nimportance-references-3d-projects-carlos-carvajal/.\nSiddhartha Chaudhuri, Evangelos Kalogerakis, Leonidas Guibas, and Vladlen Koltun. Probabilistic\nreasoning for assembly-based 3d modeling. ACM Trans. Graph., 30(4), jul 2011. ISSN 0730-\n0301.\nYiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang\nCai, Lei Yang, Gang Yu, Guosheng Lin, and Chi Zhang. Meshanything: Artist-created mesh\ngeneration with autoregressive transformers, 2024a.\nYongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu.\nComboverse:\nCompositional 3d assets creation using spatially-aware diffusion guidance. ECCV, 2024b.\nZilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion\nmodels are effective 3d generators. arXiv preprint arXiv:2403.06738, 2024c.\nYen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. Sd-\nfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the\nIEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4456–4465, 2023.\nMatt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig\nSchmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno-\ntated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern\nRecognition, pp. 13142–13153, 2023.\nLaura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann,\nThomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset\nof 3d scanned household items. In 2022 International Conference on Robotics and Automation\n(ICRA), pp. 2553–2560. IEEE, 2022.\nAnchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas O˘\nguz.\n3dgen: Triplane latent\ndiffusion for textured mesh generation. arXiv preprint arXiv:2303.05371, 2023.\nJunlin Han, Filippos Kokkinos, and Philip Torr. Vfusion3d: Learning scalable 3d generative models\nfrom video diffusion models. European Conference on Computer Vision (ECCV), 2024.\nZexin He and Tengfei Wang. Openlrm: Open-source large reconstruction models. https://\ngithub.com/3DTopia/OpenLRM, 2023.\nFangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang\nPan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion\npriors. arXiv preprint arXiv:2403.02234, 2024.\nYicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli,\nTrung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint\narXiv:2311.04400, 2023.\n11\n\n\nGabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori,\nAchal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali\nFarhadi, and Ludwig Schmidt. Openclip, July 2021.\nHeewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint\narXiv:2305.02463, 2023.\nVladimir G. Kim, Wilmot Li, Niloy J. Mitra, Siddhartha Chaudhuri, Stephen DiVerdi, and Thomas\nFunkhouser. Learning part-based templates from large collections of 3d shapes. ACM Trans.\nGraph., jul 2013.\nPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,\nHeinrich K¨\nuttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨\naschel, et al. Retrieval-augmented genera-\ntion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:\n9459–9474, 2020.\nMinghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen,\nChong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with\nconsistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.07885, 2023a.\nRuoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.\nZero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International\nConference on Computer Vision, pp. 9298–9309, 2023b.\nXiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma,\nSong-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d\nusing cross-domain diffusion. arXiv preprint arXiv:2310.15008, 2023.\nLuke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. RealFusion: 360 recon-\nstruction of any object from a single image. 2023.\nAlex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system\nfor generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.\nBen Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D\ndiffusion. In International Conference on Learning Representations (ICLR), 2023.\nRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨\norn Ommer. High-\nresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer-\nence on computer vision and pattern recognition, pp. 10684–10695, 2022.\nNadav Schor, Oren Katzir, Hao Zhang, and Daniel Cohen-Or. Componet: Learning to generate\nthe unseen by part synthesis and composition. In 2019 IEEE/CVF International Conference on\nComputer Vision (ICCV), pp. 8758–8767, 2019. doi: 10.1109/ICCV.2019.00885.\nShelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and\nYaniv Taigman.\nKnn-diffusion: Image generation via large-scale retrieval.\narXiv preprint\narXiv:2204.02849, 2022.\nRuoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen,\nChong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base\nmodel. arXiv preprint arXiv:2310.15110, 2023a.\nYichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view\ndiffusion for 3d generation. arXiv preprint arXiv:2308.16512, 2023b.\nYawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav\nRosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder-\nonly transformers. arXiv preprint arXiv:2311.15475, 2023.\nJiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm:\nLarge multi-view gaussian model for high-resolution 3d content creation.\narXiv preprint\narXiv:2402.05054, 2024.\n12\n\n\nJunshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-\nit-3d: High-fidelity 3d creation from a single image with diffusion prior.\nIn Proceedings of\nthe IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22819–22829, Octo-\nber 2023.\nVikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris-\ntian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d\ngeneration from a single image using latent video diffusion. arXiv preprint arXiv:2403.12008,\n2024.\nPeng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation.\narXiv preprint arXiv:2312.02201, 2023.\nTengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen.\nPretraining is all you need for image-to-image translation. arXiv:2205.12952, 2022.\nTengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen,\nDong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital\navatars using diffusion. In Proceedings of the IEEE/CVF conference on computer vision and\npattern recognition, pp. 4563–4573, 2023.\nZhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li,\nHang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction\nmodel. arXiv preprint arXiv:2403.05034, 2024a.\nZhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, and Rynson WH Lau. Themestation:\nGenerating theme-aware 3d assets from few exemplars. SIGGRAPH, 2024b.\nRundi Wu and Changxi Zheng.\nLearning to generate 3d shapes from a single example.\nACM\nTransactions on Graphics (TOG), 41(6), 2022.\nJiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh:\nEfficient 3d mesh generation from a single image with sparse-view large reconstruction models.\narXiv preprint arXiv:2404.07191, 2024.\nMeng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot\nnovel view synthesizer. arXiv preprint arXiv:2405.15364, 2024.\nBowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen,\nand Baining Guo. Gaussiancube: Structuring gaussian splatting using optimal transport for 3d\ngenerative modeling. arXiv preprint arXiv:2403.19655, 2024a.\nLongwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan\nXu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d\nassets, 2024b.\nLvmin Zhang, Anyi Rao, and Maneesh Agrawala.\nAdding conditional control to text-to-image\ndiffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision\n(ICCV), pp. 3836–3847, October 2023.\nJunsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d:\nExploring unified 3d representation at scale. In International Conference on Learning Represen-\ntations (ICLR), 2024.\nZi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai\nZhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction\nwith transformers. arXiv preprint arXiv:2312.09147, 2023.\n13\n\n\nAPPENDIX\nA\nIMPLEMENTATION DETAILS\nA.1\nDATASET\nTraining set. To train our reference-augmented multi-view diffusion model, we use a filtered sub-\nset of the Objaverse (Deitke et al., 2023) dataset, excluding low-quality 3D models as described\nin (Tang et al., 2024). Additionally, we apply further filtering to remove objects that are too thin and\neliminate data originating from scans, both of which are intended to ensure the quality of subsequent\nretrieval. We also exclude objects with an excessively high number of vertices or faces to optimize\nthe costly point cloud extraction process and reduce computational time. These refinements result\nin a final training set comprising approximately 64K 3D objects. For each object, we normalize\nit within a unit sphere, and render 1 concept image, 6 canonical coordinate maps (CCMs), and 6\ntarget RGBA images, following the camera distribution protocol of Zero123++ (Shi et al., 2023a).\nIn particular, the concept image is rendered using randomly sampled azimuth and elevation angles\nfrom a predefined range. The poses of the six corresponding CCMs and target images consist of\ninterleaving absolute elevations of {20°, −10°, 20°, −10°, 20°, −10°}, and relative azimuths of\n{ϕ + 30°, ϕ + 90°, ϕ + 150°, ϕ + 210°, ϕ + 270°, ϕ + 330°}, where ϕ represents the azimuth of the\nconcept image. To train our sparse-view 3D reconstruction model, we adopt the same training set\nand render images from 32 randomly sampled camera views. All images are rendered at a resolution\nof 512 × 512, a fixed absolute field of view (FOV) of 30°, and a fixed camera distance of 1.866.\nRetrieval data and method. We leverage Uni3D (Zhou et al., 2024) to retrieve a 3D reference from\nan input image. In Uni3D, the latent space of the point cloud encoder is aligned to the OpenCLIP (Il-\nharco et al., 2021) image embedding space, facilitating seamless image-to-PointCloud retrieval. Be-\nfore retrieval, point clouds are sampled from meshes according to the probability distribution of\nface areas, ensuring denser sampling in regions with larger surface areas. Each point cloud contains\n10K points. As point cloud preprocessing is time-consuming, we limit our retrieval to a subset of\n40K objects from Objaverse. Our retrieval database contains precomputed embeddings generated\nby the Uni3D point cloud encoder, which are compared with the query vector of an input image\nusing cosine similarity. To obtain the query vector, we first apply normalization transforms to align\nthe input image with the pre-trained EVA02-E-14-plus model from OpenCLIP, which acts as the\nquery encoder. The normalized image is then encoded into a feature vector. The top candidates are\nselected based on the highest similarity scores, and a softmax function is applied to the top-k scores\nto enable probabilistic sampling, ensuring efficient and accurate matching between the input image\nand the corresponding point clouds.\nA.2\nTRAINING\nReference-augmented multi-view diffusion model. White-Background Zero123++. As discussed\nin Sec. 3.1, we select Zero123++ as our initial multi-view diffusion model. Upon receiving an input\nimage, Zero123++ generates a tailored multi-view image at a resolution of 960×640, comprising six\n320×320 views arranged in a 3×2 grid. The original Zero123++ produces images with a gray back-\nground, which can result in floaters and cloud-like artifacts during the subsequent sparse-view 3D\nreconstruction phase. To mitigate this issue, we initialize our model with a variant of Zero123++ (Xu\net al., 2024), which is finetuned to generate multi-view images with a white background.\nTraining Details. During the training of our reference-augmented multi-view diffusion model, we\nuse the rendered concept image and six CCMs of a 3D object as conditions, and six corresponding\ntarget images tailored to a 960 × 640 image as ground truth image for denoising. All images and\nCCMs have a white background. We concatenate the concept image and the front-view CCM along\nthe RGB channel as the input for meta-ControlNet. For the proposed dynamic reference routing,\nwe dynamically downsample the original CCMs to lower resolutions and then upsample them to\n320 × 320, using the nearest neighbor. Specifically, we start with a resolution of 16 at noise levels\nof [0, 0.05) and gradually increase the resolution to 32 and 64 at noise levels of [0.05, 0.4) and\n[0.4, 1.0], respectively. For self-reference augmentations (Sec. A.4), the probabilities of applying\nrandom resize, flip horizontal, grid distortion, shift, and retrieved reference are set to 0.4, 0.5, 0.1,\n0.5, and 0.2, respectively. We train the model for 10,000 steps, beginning with 1000 warm-up steps\n14\n\n\nFigure 12: Detailed architecture design of meta-ControlNet.\nwith minimal augmentations. We use the AdamW optimizer with a learning rate of 1.0×10−5 and a\ntotal batch size of 48. The whole training process takes around 10 hours on 8 NVIDIA A100 (80G)\nGPUs.\nSparse-view 3D reconstruction model. As discussed in Sec. 3.5, we employ LGM to convert the\nsynthesized multi-view images into a 3D model. The original LGM is designed to reconstruct a\n3D model from four input views at a resolution of 256 × 256. However, this does not align with\nthe multi-view images generated in our first stage, which consist of six views at a resolution of\n320 × 320. To adapt LGM to our specific inputs, we take its pretrained weights as initialization\nand finetune it to support six input images at 320 × 320. Simultaneously changing the number of\ninput views and image resolutions can destabilize the training process. We therefore separate the\nfinetuning of number of input views and input resolution. Specifically, we first finetune the model\nwith six input views at the original resolution for 60 epochs and then further finetune the model at\na higher resolution of 320 × 320 for another 60 epochs. The finetuning process is conducted on 32\nNVIDIA A100 (80G) GPUs using the AdamW optimizer with a learning rate of 2.0 × 10−4 and a\ntotal batch size of 192. The whole finetuning process takes around four days.\nA.3\nMETA-CONTROLNET\nA detailed figure of the proposed meta-ControlNet in the style of vanilla ControlNet is shown\nin Fig. 12, where cpair is a pair of the concept image and the front-view reference CCM.\n15\n\n\nOurs\n3D Reference\nCRM\nLGM\nInstantMesh\nSV3D\nOpenLRM\nFrame 1\nFrame 2\nFrame 3\nFrame 4\nFigure 13: Analysis on different input viewpoints. We compare the performance of Phidias with\nfive baseline methods by reconstructing 3D objects from video frames with various viewpoints. For\neach case, we show two rendered images at novel views.\nA.4\nAUGMENTATION DETAILS\nWe implement a series of augmentations to facilitate the training of our diffusion model in a self-\nreference manner, where the ground truth 3D model serves as its own reference. These augmenta-\ntions are designed to simulate the misalignment between the 3D reference and the concept image.\nResize and horizontal flip. Due to the self-reference strategy, reference CCMs are always pixel-wise\naligned with the concept image. However, during inference, references often differ in scale or exhibit\nmirror symmetry. For example, a reference 3D character might hold a weapon in the opposite hand\ncompared to the concept image. To address this, we apply random resizing and horizontal flipping\nto the reference model, simulating scale variations and mirror-symmetric structures.\nGrid distortion and shift. During inference, the reference may exhibit asymmetric similarity with the\ntarget 3D model across different views. For instance, a reference building might closely resemble\nthe concept image from the front but differ significantly from the side. To address this, we apply\nmulti-view jitter through grid distortion and shifting. Specifically, we independently distort and shift\neach view of the reference CCMs using a random grid and a random shift offset during training,\nsimulating such asymmetric similarity across views.\nRetrieved Reference. Although the retrieved 3D reference alone is insufficient for model training, as\ndiscussed in Sec. 3.4, it can still serve as a strong augmentation to simulate significant misalignment.\nTherefore, we assign a small probability of using the retrieved model as the reference during training.\n16\n\n\nGenerated 3D Model\nInput Image\n3D Ref. CCM\nGenerated 3D Model\nInput Image\n3D Ref. CCM\n(a) Angle deviation between input image and 3D reference\n(b) Semantic-aligned but structural-misaligned 3D reference\n(30°, 20°)\n(90°, −10°)\n(150°, 20°)\n(210°, −10°)\n(30°, 20°)\n(90°, −10°)\n(150°, 20°)\n(210°, −10°)\nFigure 14: Failure cases. There are two typical failure cases due to bad retrieval: (a) misaligned\npose and (b) misaligned structure.\nB\nLIMITATION AND FAILURE CASES\nDespite promising results, Phidias still has several limitations for further improvement.\nAs a\nretrieval-augmented generation model, the performance can be affected by the retrieval method and\nthe scale and quality of 3D reference database. Currently, the 3D database we used for retrieval\nonly consists of 40K objects, making it difficult to find a very similar match. Also, mainstream\n3D retrieval methods rely on semantic similarity, which may not always yield the best match. For\nexample, retrieved reference models with misaligned poses or structures can lead to undesired out-\ncomes, as shown in Fig. 14. Future works that improve the retrieval accuracy and expand the 3D\nreference database could mitigate these issues. Additionally, the limited resolution of the backbone\nmulti-view diffusion model (320×320) restricts the handling of high-resolution images. Enhancing\nthe resolution of the diffusion model could further improve the quality of the generated 3D models.\nC\nADDITIONAL RESULTS\nC.1\nADDITIONAL ANALYSIS ON ENHANCED GENERALIZATION ABILITY\nPhidias takes an additional 3D reference as input to improve generative quality (Fig. 5) and provide\ngreater controllability (Fig. 4) for 3D generation. We argue that Phidias can also enhance general-\nization ability when given input images from atypical viewpoints. When reconstructing 3D objects\nfrom video frames with varying views (Fig. 13), we observe that the baseline methods perform well\nwith typical view angles (i.e., frame 1) but struggle with atypical input view angles (e.g., frame 3 and\n4). Conversely, Phidias produces plausible results given all four input views, demonstrating robust\ngeneralization ability across both typical and atypical viewpoints.\nC.2\nMORE RESULTS\nMore results on theme-aware 3D-to-3D generation are shown in Fig. 15. More results on text-to-3D\nand image-to-3D generation are shown in Fig. 16 and Fig. 17.\n17\n\n\n3D Input\nSelf-Reference\nGenerated 3D Variation 1\nGenerated 3D Variation 2\nFigure 15: Additional results on theme-aware 3D-to-3D generation.\n18\n\n\nText Input\nGenerated 3D Model\n3D Reference\n“Glowing \nmushroom forest \nwith stars”\n“Red and silver \nmotorcycle”\nText Input\nGenerated 3D Model\n3D Reference\n“Golden and silver \nmedieval knight's \nhelmet”\n“Green and \nyellow ceramic \nincense vessel”\n“Blue armored \nrobot with angular \ndesign”\n“Bulky robot with \ntwo mechanical \narms”\nFigure 16: Additional results on retrieval-augmented text-to-3D generation.\nImage Input\nGenerated 3D Model\n3D Reference\nImage Input\nGenerated 3D Model\n3D Reference\nFigure 17: Additional results on retrieval-augmented image-to-3D generation.\n19\n\n\nUnder review as a conference paper at ICLR 2024\nDMV3D: DENOISING MULTI-VIEW DIFFUSION USING\n3D LARGE RECONSTRUCTION MODEL\nAnonymous authors\nPaper under double-blind review\nABSTRACT\nWe propose DMV3D, a novel 3D generation approach that uses a transformer-\nbased 3D large reconstruction model to denoise multi-view diffusion. Our re-\nconstruction model incorporates a triplane NeRF representation and, functioning\nas a denoiser, can denoise noisy multi-view images via 3D NeRF reconstruction\nand rendering, achieving single-stage 3D generation in the 2D diffusion denoising\nprocess. We train DMV3D on large-scale multi-view image datasets of extremely\ndiverse objects using only image reconstruction losses, without accessing 3D\nassets. We demonstrate state-of-the-art results for the single-image reconstruction\nproblem where probabilistic modeling of unseen object parts is required for\ngenerating diverse reconstructions with sharp textures. We also show high-quality\ntext-to-3D generation results outperforming previous 3D diffusion models. Our\nproject website is at: https://dmv3d.github.io/.\n1\nINTRODUCTION\nThe advancements in 2D diffusion models (Ho et al., 2020; Song et al., 2020a; Rombach et al.,\n2022) have greatly simplified the image content creation process and revolutionized 2D design\nworkflows. Recently, diffusion models have also been extended for 3D asset creation, which is still\na time-consuming manual task but critical for various 3D applications such as VR, AR, robotics,\nand gaming. In particular, many works have explored using pre-trained 2D diffusion models for\ngenerating NeRFs (Mildenhall et al., 2020) with score distillation sampling (SDS) loss (Poole et al.,\n2022; Lin et al., 2023a). However, SDS-based methods require long (often hours of) per-asset\noptimization and can frequently lead to rendering artifacts, such as the multi-face Janus problem.\nOn the other hand, attempts to train 3D diffusion models have also been made to enable 3D\ngeneration without per-asset optimization (Nichol et al., 2022; Jun & Nichol, 2023). These methods\ntypically include pre-training per-asset NeRFs, followed by training diffusion models on the NeRF\nlatents. However, this disjoint two-stage training, with independently trained NeRFs, often leads to\nan unclean and hard-to-denoise latent space (Chen et al., 2023), making high-quality rendering a\nchallenge. To circumvent this, single-stage models have been proposed (Anciukeviˇ\ncius et al., 2023;\nKarnewar et al., 2023), but are all category-specific and unable to generalize beyond simple classes.\nOur goal is to achieve fast, realistic, and generic 3D generation. To this end, we propose DMV3D,\na novel single-stage category-agnostic diffusion model that can generate 3D (triplane) NeRFs from\ntext or single-image input conditions via direct model inference. Our model allows for the generation\nof diverse high-fidelity 3D objects within one minute per asset (see Fig. 1). In particular, DMV3D is\na 2D multi-view image diffusion model that integrates 3D NeRF reconstruction and rendering into\nits denoiser, trained without direct 3D supervision, in an end-to-end manner. This avoids both pre-\ntraining 3D NeRFs (as in two-stage models) and tedious per-asset optimization (as in SDS methods).\nIn essence, our approach jointly addresses 2D image (diffusion) denoising and 3D reconstruction.\nThis is inspired by RenderDiffusion (Anciukeviˇ\ncius et al., 2023) – achieving 3D generation through\nsingle-view diffusion. However, their single-view framework relies on category-specific priors and\ncanonical poses and thus cannot easily be scaled up to generate arbitrary objects. In contrast, we\nconsider a sparse set of four multi-view images that surround an object, adequately expressing a full\n3D asset. This design choice is inspired by humans, who can easily imagine a complete 3D object\nfrom a few surrounding views with little uncertainty. However, utilizing such inputs essentially\n1\n\n\nUnder review as a conference paper at ICLR 2024\nFigure 1: Top left: our approach achieves fast 3D generation from text or single-image input; the\nlatter one, combined with 2D segmentation methods (like SAM (Kirillov et al., 2023)), allows us to\nreconstruct objects segmented from natural images. Bottom: as a probabilistic generative model, our\nmodel can produce multiple reasonable 3D assets from the same image. Top right: we demonstrate\na scene comprising diverse 3D objects generated by our models, each within one minute.\nrequires addressing the task of sparse-view 3D reconstruction – a long-standing problem and known\nto be highly challenging even without noise in the inputs.\nWe address this by leveraging the power of large transformer models that have been shown to be\neffective and scalable in solving language and multi-modal problems. Specifically, we propose a\nnovel transformer-based large 3D reconstruction model that can, from a sparse set of noisy multi-\nview images, reconstruct a clean (noise-free) NeRF model that allows for rendering (denoised)\nimages at arbitrary viewpoints. Our transformer model is conditioned on the diffusion time step,\ndesigned to handle any noise levels in the diffusion process. It can thus be directly plugged as the\nmulti-view image denoiser in an multi-view image diffusion framework.\nMoreover, the nature of being a 2D diffusion model allows for natural inheritance of the succeses in\nexiting 2D diffusion models, including the ability to handle various input conditions. In particular,\nwe enable single-image conditioning by simply fixing one of the sparse views as the noise-free\ninput and denoising other views, posing the task as one similar to (multi-view) image inpainting.\nIn addition, we apply attention-based text conditioning and classifier-free guidance, widely used in\n2D diffusion models, to enable text-to-3D generation. We train our model on large-scale datasets of\nboth synthetic renderings and real captures with purely multi-view image supervision. Our model\nachieves state-of-the-art results on single-image 3D reconstruction on multiple testing datasets,\noutperforming both SDS-based methods and 3D diffusion models. We also demonstrate high-quality\ntext-to-3D results outperforming previous 3D diffusion models. In sum, our main contributions are:\n• A novel single-stage diffusion framework that leverages multi-view 2D image diffusion\nmodel to achieve 3D generation;\n• A novel transformer-based large reconstruction model that can reconstruct noise-free\ntriplane NeRFs from noisy multi-view images;\n• A general approach for high-quality text-to-3D generation and single-image reconstruction.\nOur work offers a novel perspective to address 3D generation tasks, which bridges 2D and 3D\ngenerative models and unifies 3D reconstruction and generation. This opens up opportunities to\nbuild a foundation model for tackling a variety of 3D vision and graphics problems.\n2\n\n\nUnder review as a conference paper at ICLR 2024\nFigure 2: Single-image reconstruction with SAM. We can use SAM (Kirillov et al., 2023) to\nsegment any objects from a real photo and reconstruct their 3D shape and appearance with our\nmethod, demonstrating the robustness and generalizability of our method.\n2\nRELATED WORK\nSparse-view Reconstruction. Neural representations (Mescheder et al., 2019; Park et al., 2019;\nMildenhall et al., 2020; Sitzmann et al., 2019; 2020; Chen et al., 2022; M¨\nuller et al., 2022) offer\na promising platform for scene representation and neural rendering (Tewari et al., 2022). Applied\nto novel-view synthesis, these approaches have been successful in single-scene overfitting scenarios\nwhere lots of multi-view training images are available. Recent efforts (Yu et al., 2021; Chen et al.,\n2021; Long et al., 2022; Wang et al., 2021; Lin et al., 2023b; Jain et al., 2021) have extended\nthese ideas to operate with a sparse set of views, showcasing improved generalization capabilities\nto unseen scenes. As non-generative methods, however, these approaches struggle on attempting to\nscale learning up to large datasets and they exhibit limited performance on diverse data.\n3D Generative Adversarial Networks (GANs). GANs have made remarkable advancements in\n2D image synthesis (Brock et al., 2018; Karras et al., 2018; 2019; 2020; 2021). 3D GANs (Nguyen-\nPhuoc et al., 2019; Schwarz et al., 2020; Chan et al., 2021; 2022; Niemeyer & Geiger, 2021;\nGu et al., 2021; Skorokhodov et al., 2022; Xu et al., 2022; 2023; Shi et al., 2022; Gao et al.,\n2022; Skorokhodov et al., 2023) extend these capabilities to generating 3D-aware assets from\nunstructured collections of single-view 2D images in an unsupervised manner. GAN architectures,\nhowever, are difficult to train and generally best suited for modeling datasets of limited scale and\ndiversity (Dhariwal & Nichol, 2021).\n3D-aware Diffusion Models (DMs).\nDMs have emerged as foundation models for visual\ncomputing, offering unprecedented quality, fine-grained control, and versatility for 2D image\ngeneration (Ho et al., 2020; Song et al., 2020a;b; Rombach et al., 2022). Several strategies have been\nproposed to extend DMs to the 3D domain. Some of these approaches (Jun & Nichol, 2023; Shue\net al., 2023; Nichol et al., 2022; Gupta et al., 2023; Ntavelis et al., 2023) use direct 3D supervision.\nThe quality and diversity of their results, however, is far from that achieved by 2D DMs. This\nis partly due to the computational challenge of scaling diffusion network models up from 2D to\n3D, but perhaps more so by the limited amount of available 3D training data. Other approaches\nin this category build on optimization using a differentiable 3D scene representation along with\nthe priors encoded in 2D DMs (Poole et al., 2022; Lin et al., 2023a; Wang et al., 2022; 2023).\nWhile showing some success, the quality and diversity of their results is limited by the SDS–based\nloss function (Poole et al., 2022). Another class of methods uses 2D DM–based image-to-image\ntranslation using view conditioning (Liu et al., 2023b; Chan et al., 2023; Gu et al., 2023). While\nthese approaches promote multi-view consistency, they do not enforce it, leading to flicker and other\nview-inconsistent effects. Finally, several recent works have shown success in training 3D diffusion\nmodels directly on multi-view image datasets (Karnewar et al., 2023; Chen et al., 2023) for relatively\nsimple scenes with limited diversity.\nRenderDiffusion (Anciukeviˇ\ncius et al., 2023) and its successor Viewset Diffusion (Szymanowicz\net al., 2023), which is concurrent to this work, are closest to our method. Both solve the sparse-\nview reconstruction problem using 2D DMs with 3D-aware denoisers. Neither of these methods,\nhowever, has been demonstrated to work on extremely diverse datasets containing multi-view data\nof >1M objects. Our novel transformer-based 3D denoiser architecture overcomes this challenge\nand enables state-of-the-art results for scalable, diverse, and high-quality 3D generation.\n3\n\n\nUnder review as a conference paper at ICLR 2024\n© 2023 Adobe. All Rights Reserved. Adobe Confid\nImage \ntokenizer \n(DINO)\nReshape & \nUpsample\nt\nImage tokens\nTransformer\nCross-Att\nMLP\n+\n+\nSelf-Att\n+\nText\nt\nTriplane position \nembeddings\nPlücker rays\nTriplane \ntokens\nt-1\nRendering loss\nFigure 3: Overview of our method. We denoise multiple views (three shown in the figure; four\nused in experiments) for 3D generation. Our multi-view denoiser is a large transformer model that\nreconstructs a noise-free triplane NeRF from input noisy images with camera poses (parameterized\nby Plucker rays). During training, we supervise the triplane NeRF with a rendering loss at input and\nnovel viewpoints. During inference, we render denoised images at input viewpoints and combine\nthem with noise to obtain less noisy input for the next denoising step. Once the multi-view images\nare fully denoised, our model offers a clean triplane NeRF, enabling 3D generation. Refer to Sec. 3.3\nfor how to extend this model to condition on single image.\n3\nMETHOD\nWe now present our single-stage diffusion model. In particular, we introduce a novel diffusion\nframework that uses a reconstruction-based denoiser to denoise multi-view noisy images for 3D\ngeneration (Sec. 3.1). Based on this, we propose a novel large 3D reconstruction model conditioned\non diffusion time step, functioning as the multi-view denoiser, to denoise multi-view images via\n3D NeRF reconstruction and rendering (Sec. 3.2). We further extend our model to support text and\nimage conditioning, enabling practical and controllable generation (Sec. 3.3).\n3.1\nMULTI-VIEW DIFFUSION AND DENOISING\nDiffusion. Denoising Diffusion Problistic Models (DDPM) extends the data distribution x0 ∼\nq(x) with a T-step Markov Chain using a Gaussian noise schedule. The generation process is\nthe reverse of a forward diffusion process. The diffusion data xt at timestep t can be derived by\nxt = √¯\nαtx0 + √1 −¯\nαtϵ, where ϵ ∼N(0, I) represents Gaussian noise and ¯\nαt is a monotonically\ndecreasing noise schedule.\nMulti-view diffusion. The original x0 distribution addressed in 2D DMs is the (single) image\ndistribution in a dataset.\nWe instead consider the (joint) distribution of multi-view images\nI = {I1, ..., IN}, where each set of I are image observations of the same 3D scene (asset)\nfrom viewpoints C = {c1, ..., cN}. The diffusion process is equivalent to diffusing each image\nindependently but with the same noise schedule:\nIt = {√¯\nαtI +\n√\n1 −¯\nαtϵI|I ∈I}\n(1)\nNote that this diffusion process is identical to the original one in DDPM, despite that we consider a\nspecific type of data distribution x = I of per-asset 2D multi-view images.\nReconstruction-based denoising. The reverse of the 2D diffusion process is essentially denoising.\nIn this work, we propose to leverage 3D reconstruction and rendering to achieve 2D multi-view\nimage denoising, while outputting a clean 3D model for 3D generation. In particular, we leverage\na 3D reconstruction module E(·) to reconstruct a 3D representation S from the noisy multi-view\nimages It (at time step t), and render denoised images with a differentiable rendering module R(·):\nIr,t = R(St, c),\nSt = E(It, t, C)\n(2)\nwhere Ir,t represents a rendered image from St at a specific viewpoint c.\n4\n\n\nUnder review as a conference paper at ICLR 2024\nDenoising the multi-view input It is done by rendering St at the viewpoints C, leading to the\nprediction of noise-free I0. This is equivalent to x0 prediction in 2D DMs (Song et al., 2020a),\nwhich can be used to predict xt−1, enabling progressive denoising inference. However, unlike\npure 2D generation, we find merely supervising I0 prediction at input viewpoints cannot guarantee\nhigh-quality 3D generation (see Tab. 3), often leading to rendering artifacts at novel viewpoints.\nTherefore, we propose to also supervise images rendered at novel viewpoints from the 3D model St.\nIn essence, we reposition the original 2D image x0 (I0) prediction to a (hidden) 3D S0 prediction\ntask, ensuring consistent high-quality rendering across arbitrary viewpoints. The denoising objective\nis written as\nLrecon(t) = EI,c∼Ifull,Cfull∥I −R(E(It, t, C), c)∥2\n2\n(3)\nwhere Ifull and Cfull represent the full set of images and poses (from both input and novel views).\nNote that our framework is general – potentially any 3D representations (S) can be applied. In\nthis work, we consider a (triplane) NeRF representation (where R(·) becomes neural volumetric\nrendering) and propose a transformer-based reconstructor E(·).\n3.2\nRECONSTRUCTOR-BASED MULTI-VIEW DENOISER\nWe seek to build a robust reconstructor that can recover 3D shape and appearance from sparse multi-\nview images. As in previous work (Chan et al., 2022), we adopt the triplane NeRF as a compact\nand efficient 3D representation. However, in contrast to previous work that relies on CNNs, we use\na transformer-based large reconstruction model that, given 2D image tokens and learnable triplane\ntokens, effectively reconstructs a 3D NeRF model that supports realistic rendering.\nReconstruction and rendering. As shown in Fig. 3, we tokenize the triplane with learnable tokens\n(T) and use a Vision Transformer (DINO) to convert input images I = {I1, ..., IN} (N = 4 by\ndefault) to 2D tokens. We apply a large transformer model with a series of image-to-triplane cross-\nattention and triplane-to-triplane self-attention layers to regress the final tri-plane S that represents\nthe 3D shape and appearance of the asset. The triplane is then used to decode volume density\nand color with an MLP for differentiable volume rendering. In essence, this process realizes the\nEqn. 2 with a large transformer model E and neural rendering module R. Overall, our transformer\nis inspired by the large reconstruction models in Anonymous (2023a;b) and we further enable time\nconditioning for diffusion denoising and introduce a new technique for camera conditioning.\nTime Conditioning. Our transformer-based model requires different designs for time-conditioning,\ncompared to DDPM and its variants that are based on CNN UNets. Inspired by DiT (Peebles & Xie,\n2022), we apply time condition through the adaLN-Zero block in our self- and cross- attention layers\nin our model, allowing our model to effectively handle input with different diffusion noise levels.\nCamera Conditioning. Addressing sparse multi-view reconstruction requires an effective design\nof input camera conditioning for the model to understand the multi-view input and build corre-\nspondence for 3D reasoning.\nA basic strategy is, as in the case of time conditioning, to use\nadaLN-Zero block on the camera parameters (as done in Anonymous (2023b)). However, we find\nthat conditioning on camera and time simultaneously with the same strategy tends to weaken the\neffects of these two conditions and often leads to an unstable training process and slow convergence.\nInstead, we propose a novel approach – parameterizing cameras with sets of pixel-aligned rays. In\nparticular, following LFN (Sitzmann et al., 2021), we parameterize rays using Plucker coordinates\nas r = (o × d, d), where o and d are the origin and direction of a pixel ray and can be computed\nfrom the camera parameters. We concatenate the Plucker coordinates with image pixels, and send\nthem to the ViT transformer for 2D image tokenization, achieving effective camera conditioning.\n3.3\nCONDITIONING ON SINGLE IMAGE OR TEXT\nThe methods described thus far enable our model to function as an unconditional generative model.\nWe now introduce how to model the conditional probabilistic distribution with a conditional denoiser\nE(It, t, C, y), where y is text and image conditioning, enabling controllable 3D generation.\nImage Conditioning. Unlike previous methods (Liu et al., 2023b) that design new modules to\ninject image conditioning to a DM, we propose a simple but effective view-inpainting strategy for\nour multi-view model. In particular, we keep the first view I1 (in the denoiser input) noise-free as the\n5\n\n\nUnder review as a conference paper at ICLR 2024\nimage condition, while applying diffusion and denoising on other views. In this case, the denoiser\nessentially learns to fill in the missing pixels within the noisy views using cues extracted from the\nfirst input view, similar to the task of image inpainting which has been shown to be addressable\nby 2D DMs (Rombach et al., 2022). In addition, to improve the generalizability of our image-\nconditioned model, we generate tri-planes in a coordinate frame aligned with the conditional view\nand render other images using poses relative to the conditional one.\nText Conditioning. To add text conditioning into our model, we adopt a strategy similar to that\npresented in Stable Diffusion (Rombach et al., 2022). We use the text encoder from CLIP (Radford\net al., 2021) to generate text embeddings and inject them into our denoiser using cross-attention.\nSpecifically, we include an additional cross-attention layer after each self-attention block in the ViT\nand each cross-attention blocak in the triplane transformer, enabling text-driven 3D generation.\n3.4\nTRAINING AND INFERENCE\nTraining. During the training phase, we uniformly sample time steps t within the range [1, T],\nand add noise according to a cosine schedule.\nWe sample input images with random camera\nposes, instead of fixing ones, enhancing the robustness of our system. We also randomly sample\nadditional novel viewpoints to supervise the renderings (as discussed in Sec. 3.1) for better quality.\nWe minimize the following training objective with conditional signal y:\nL = Et∼U[1,T ],(I,c)∼(Ifull,Cfull)∥I −R(E(It, t, D, y), c)∥2\n2\n(4)\nInference. For inference, we select four viewpoints that uniformly surround the object in a circle\nwith the same pitch, to ensure the reconstruction model (denoiser) can capture the full 3D shape and\nappearance. We utilize DDIM (Song et al., 2020a) to improve the inference speed in the progressive\nmulti-view denoising. Once the 2D multi-view images are fully denoised at the final step, we can\ndirectly obtain a clean triplane NeRF model from the denoiser, achieving fast 3D generation without\nrequiring any extra optimization to fit the multi-view denoised images.\n4\nEXPERIMENTS\nIn this section, we present an extensive evaluation of our method. In particular, we briefly describe\nour experiment settings (Sec. 4.1), compare our results with previous works (Sec. 4.2), and show\nadditional analysis and ablation experiments (Sec. 4.3).\n4.1\nSETTINGS\nImplementation details. We use Adam optimizer to train our model with an initial learning rate of\n4e−4. We also apply a warm-up stage for 3K steps and a cosine decay on the learning rate. We train\nour denoiser with 256 × 256 input images and render 128 × 128 image crops for supervision. Our\nfinal model is a large transformer with 48 attention layers and 643 triplane tokens with 32 channels.\nWe use 128 NVIDIA A100 GPUs to train this model with a batch size of 8 per GPU for 100K steps,\ntaking about 7 days. Since the final model takes a lot of resources, it is impractical for us to evaluate\nthe design choices with this large model for our ablation study. Therefore, we also train a small\nmodel that consists of 36 attention layers to conduct our ablation study. The small model is trained\nwith 32 NVIDIA A100 GPUs for 200K steps (4 days).\nDatasets. Our model requires only 2D image supervision. We use rendered multi-view images from\n∼700k scenes in the Objaverse (Deitke et al., 2023) dataset to train our text-to-3D model, for which\nwe use Cap3D (Luo et al., 2023) to generate the text prompts. For each scene, we render 32 images\nunder uniform lighting at random viewpoints with a fixed 50◦FOV. For image-conditioned (single-\nview reconstruction) model, we combine the Objaverse data with additional real captures of ∼200k\nscenes from the MVImgNet (Yu et al., 2023) dataset, enhancing the generalization to out-of-domain\ninput (see Fig. 7). In general, these datasets contain a large variety of synthetic and real assets from\nnumerous categories, allowing us to train a generic and scalable 3D generative model.\nWe evaluate our image-conditioned model with novel synthetic datasets, including 100 scenes from\nthe Google Scanned Object (GSO) (Downs et al., 2022) and 100 scenes from the Amazon Berkeley\nObject (ABO) (Collins et al., 2022) datasets. This allows for direct comparison of single-view\n6\n\n\nUnder review as a conference paper at ICLR 2024\nTable 1: Evaluation Metrics of single-image 3D reconstruction on ABO and GSO datasets.\nABO dataset\nGSO dataset\nFID ↓\nCLIP ↑\nPSNR ↑\nLPIPS ↓\nCD ↓\nFID ↓\nCLIP ↑\nPSNR ↑\nLPIPS ↓\nCD ↓\nPoint-E\n112.29\n0.806\n17.03\n0.363\n0.127\n123.70\n0.741\n15.60\n0.308\n0.099\nShap-E\n79.80\n0.864\n15.29\n0.331\n0.097\n97.05\n0.805\n14.36\n0.289\n0.085\nZero123\n31.59\n0.927\n17.33\n0.194\n−\n32.44\n0.896\n17.36\n0.182\n−\nOne2345\n190.81\n0.748\n12.00\n0.514\n0.163\n139.24\n0.713\n12.42\n0.448\n0.123\nMagic123\n34.93\n0.928\n18.47\n0.180\n0.136\n34.06\n0.901\n18.68\n0.159\n0.113\nOurs (S)\n36.77\n0.915\n22.62\n0.194\n0.059\n35.16\n0.888\n21.80\n0.150\n0.046\nOurs\n27.88\n0.949\n24.15\n0.127\n0.046\n30.01\n0.928\n22.57\n0.126\n0.040\nOurs\nInput\nShapE\nPointE\nOne2345\nMagic123\nFigure 4: Qualitative comparisons on single-image reconstruction.\nreconstruction with the groundtruth. Note that accurate quantitative evaluation of 3D generation\nremains a challenge in the field, we use the most applicable metrics from earlier works to assess our\nand baseline models.\n4.2\nRESULTS AND COMPARISONS\nSingle-image reconstruction. We compare our image-conditioned model with previous methods,\nincluding Point-E (Nichol et al., 2022), Shap-E (Jun & Nichol, 2023), Zero123 (Liu et al., 2023b),\nOne2345 (Liu et al., 2023a), and Magic123 (Qian et al., 2023), on single-image reconstruction. We\nevaluate the novel-view rendering quality from all methods using PSNR, LPIPS, CLIP precision\n(including top-1 R-precision and averaged precision), and FID, computed between the rendered and\nGT images. In addition, we also compute the Chamfer distance (CD) for geometry evaluation, for\nwhich we use marching cubes to extract meshes from NeRFs.\nTable 1 report the quantitative results on the GSO and ABO testing sets respectively. Note that our\nmodels (even ours-small ) can outperforms all baseline methods, achieving the best scores across all\n7\n\n\nUnder review as a conference paper at ICLR 2024\nOurs\nShap-E\nPoint-E\n‘a bowl of vegetables'\n‘a voxelized dog'\n‘a rusty old car'\nFigure 5: Qualitative comparison on Text-to-3D .\nmetrics for both datasets. Our high generation quality is reflected by the qualitative results shown\nin Fig. 4; our model generates realistic results with more complete geometry and much sharper\nappearance details, compared to all baselines.\nTable 2: Evaluation Metrics on Text-to-3D.\nMethod\nVIT-B/32\nViT-L/14\nR-Prec\nAP\nR-Prec\nAP\nPoint-E\n33.33\n40.06\n46.4\n54.13\nShap-E\n38.39\n46.02\n51.40\n58.03\nOurs\n39.72\n47.96\n55.14\n61.32\nIn particular, the two-stage 3D DMs, ShapE and\nPoint-E, lead to lower quality, often with incomplete\nshapes and blurry textures; this suggests the inherent\ndifficulties in denoising pretrained 3D latent spaces,\na problem our model avoids.\nOn the other hand,\nZero123 leads to better quantitative results than ShapE\nand Point-E on appearnce, because it is a 2D diffusion\nmodel and trained to generate high-quality images.\nHowever, Zero123 alone cannot output a 3D model\nrequired by many 3D applications and their rendered\nimages suffer from severe inconsistency across viewpoints. This inconsistency also leads to the\nlow reconstruction and rendering quality from One2345, which attempts to reconstruct meshes from\nZero123’s image outputs. On the other hand, the per-asset optimization-based method Magic123\ncan achieve rendering quality comparable to Zero123 while offering a 3D mdoel. However, these\nmethods require long (hours of) optimization time and also often suffer from unrealistic Janus\nartifacts (as shown in the second object in Fig. 4). In contrast, our approach is a single-stage\nmodel with 2D image training objectives and directly generates a 3D NeRF model (without per-\nasset optimization) while denoising multi-view diffusion. Our scalable model learns strong data\npriors from massive training data and produces realistic 3D assets without Janus artifacts. In general,\nour approach leads to fast 3D generation and state-of-the-art single-image 3D reconstruction results.\nTable 3: Ablation on GSO dataset (DMV3D-S). See\nFig. 8 for qualitative results.\n#Views\nFID ↓\nCLIP ↑\nPSNR ↑\nSSIM ↑\nLPIPS ↓\nCD ↓\n4 (Ours)\n35.16\n0.888\n21.798\n0.852\n0.150\n0.0459\n1\n70.59\n0.788\n17.560\n0.832\n0.304\n0.0775\n2\n47.69\n0.896\n20.965\n0.851\n0.167\n0.0544\n6\n39.11\n0.899\n21.545\n0.861\n0.148\n0.0454\nw.o Novel\n102.00\n0.801\n17.772\n0.838\n0.289\n0.185\nw.o Plucker\n43.31\n0.883\n20.930\n0.842\n0.185\n0.505\nText-to-3D. We also evaluate our text-\nto-3D generation results and compare\nwith 3D diffusion models Shap-E (Jun\n& Nichol, 2023) and Point-E (Nichol\net al., 2022), that are also category-\nagnostic and support fast direct infer-\nence. For this experiment, we use Shap-\nE’s 50 text prompts for the generation,\nand evaluate the results with CLIP pre-\ncisions using two different ViT models,\nshown in Table. 2. From the table, we\ncan see that our model achieves the best precision. We also show qualitative results in Fig. 5, in\nwhich our results clearly contain more geometry and appearance details and look more realistic than\nthe compared ones.\n4.3\nANALYSIS, ABLATION, AND APPLICATION\nWe analyze our image-conditioned model and verify our design choices using our small model\narchitecture for better energy-efficiency.\n8\n\n\nUnder review as a conference paper at ICLR 2024\nInput\nNovel-view\nInput\nNovel-view\nFigure 6: Robustness on out-of-domain inputs of synthetic, real, and generated images.\n#Views. We show quantitative and qualitative comparisons of our models trained with different\nnumbers (1, 2, 4, 6) of input views in Tab. 3 and Fig. 8. We can see that our model consistently\nachieves better quality when using more images, benefiting from capturing more shape and\nappearance information. However, the performance improvement of 6 views over four views is\nmarginal, where some metrics (like PSNR) from the 4-view model is even better. We therefore use\nfour views as the default setting to generate all of our main results.\nMultiple inference generation. Similar to other DMs, our model can generate various instances\nfrom the same input image with different random seeds as shown in Fig. 1, demonstrating the\ndiversity of our generation results. In general, we find the multiple inference results can all reproduce\nthe frontal input view while containing varying shape and appearance in the unseen back side.\nInput sources. Our model is category-agnostic and generally works on various input sources as\nshown in many previous figures. We show additional results in Fig. 6 with various inputs, out of our\ntraining domains, including synthetic rendering, real capture, and generated images. Our method\ncan robustly reconstruct the geometry and appearance of all cases.\nTraining data. We compare our models trained w/ and w.o the real MVImgNet dataset on two\nchallenging examples. As shown in Fig. 7, we can see that the model without MVImgNet can lead\nto unrealistic flat shapes, showcasing the importance of having diverse training data.\nMore ablation. We compare with our ablated models including one trained without the novel-view\nrendering supervision, and one without the Plucker coordinate view conditioning (using the adaLN-\nZero block conditioning instead). We can also see that the novel view rendering supervision is critical\nfor our model. Without it, all quantitative scores drop by a large margin. In general, the novel view\nsupervision is crucial for our model to achieve meaningful 3D generation, avoiding the model to\nlearn a local minima that merely recovers the sparse multi-view images. In addition, our design of\nPlucker coordinate-based camera conditioning is also effective, leading to better quantitative results\nthan the ablated model.\nApplication.\nThe flexibility and generality of our method can potentially enable broad 3D\napplications. One useful image editing application is to lift any objects in a 2D photo to 3D by\nsegment them (using methods like SAM (Kirillov et al., 2023)) and reconstruct the 3D model with\nour method, as shown in Fig. 1 and 2.\n5\nCONCLUSION\nWe presented a novel single-stage diffusion model for 3D generation, generating 3D assets by\ndenoising multi-view image diffusion. Our multi-view denoiser is based on a large transformer\nmodel, which takes multi-view noisy images to reconstruct a clean triplane NeRF, outputting\ndenoised images through neural rendering. Our framework generally supports text- and image-\nconditioning inputs, achieving fast 3D generation via direct diffusion inference without per-asset\noptimization. Our method outperforms previous 3D diffusion models for text-to-3D generation\nand achieves state-of-the-art quality on single-view reconstruction on various testing datasets. Our\napproach combines 2D diffusion and 3D reconstruction, bridging the gap between 2D and 3D\ngeneration and paving the way for future directions on extending 2D diffusion applications for 3D\ngeneration.\n9\n\n\nUnder review as a conference paper at ICLR 2024\nEthics Statement.\nOur generative model is trained on the Objaverse data and MvImgNet data.\nThe dataset (about 1M) is smaller than the dataset in training 2D diffusion models (about 100M\nto 1000M). The lack of data can raise two considerations. First, it can possibly bias towards the\ntraining data distribution. Secondly, it might not be powerful enough to cover all the diversity of\ntesting images and testing texts. Our model has certain generalization ability but might not cover\nas much modes as the 2D diffusion model can. Given that our model does not have the ability to\nidentify the content that is out of its knowledge, it might introduce to unsatisfying user experience.\nAlso, our model can possibly leak the training data if the text prompt or image input highly align\nwith some data sample. This potential leakage raises legal and security considerations, and is shared\namong all generative data (such as LLM and 2D diffusion models).\nReproducibility Statement.\nWe provide detailed implementation of our training method in the\nmain text and also provide the model configurations in Table 6.\nWe will help to resolve any\nuncertainty of our implementation during review discussions.\nREFERENCES\nTitas Anciukeviˇ\ncius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and\nPaul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation.\nIn IEEE Conf. Comput. Vis. Pattern Recog., 2023.\nAnonymous. Lrm: Large reconstruction model for single image to 3d. In Supplementary Files,\n2023a.\nAnonymous. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.\nIn Supplementary Files, 2023b.\nAndrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural\nimage synthesis. arXiv preprint arXiv:1809.11096, 2018.\nEric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic\nimplicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conf. Comput.\nVis. Pattern Recog., 2021.\nEric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio\nGallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d\ngenerative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.\nEric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy,\nMiika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view\nsynthesis with 3d-aware diffusion models. Int. Conf. Comput. Vis., 2023.\nAnpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.\nMvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Int. Conf.\nComput. Vis., 2021.\nAnpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance\nfields. In European Conference on Computer Vision (ECCV), 2022.\nHansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-\nstage diffusion nerf: A unified approach to 3d generation and reconstruction. arXiv preprint\narXiv:2304.06714, 2023.\nJasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu,\nXi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and\nbenchmarks for real-world 3d object understanding. In IEEE Conf. Comput. Vis. Pattern Recog.,\npp. 21126–21136, 2022.\nMatt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig\nSchmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.\nObjaverse: A universe of\nannotated 3d objects. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 13142–13153, 2023.\n10\n\n\nUnder review as a conference paper at ICLR 2024\nPrafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances\nin neural information processing systems, 34:8780–8794, 2021.\nLaura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann,\nThomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset\nof 3d scanned household items. In 2022 International Conference on Robotics and Automation\n(ICRA), pp. 2553–2560. IEEE, 2022.\nJun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan\nGojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned\nfrom images. Adv. Neural Inform. Process. Syst., 2022.\nJiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware\ngenerator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021.\nJiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and\nRavi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from\n3d-aware diffusion. In Int. Conf. Mach. Learn., 2023.\nAnchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas O˘\nguz.\n3dgen: Triplane latent\ndiffusion for textured mesh generation. arXiv preprint arXiv:2303.05371, 2023.\nJonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adv. Neural\nInform. Process. Syst., 2020.\nAjay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent\nfew-shot view synthesis. In Int. Conf. Comput. Vis., 2021.\nHeewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint\narXiv:2305.02463, 2023.\nAnimesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra. Holodiffusion: Training a\n3d diffusion model using 2d images. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.\nTero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.\nProgressive growing of gans for\nimproved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018.\nTero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative\nadversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.\nTero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.\nAnalyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern\nRecog., 2020.\nTero Karras, Miika Aittala, Samuli Laine, Erik H¨\nark¨\nonen, Janne Hellsten, Jaakko Lehtinen, and\nTimo Aila. Alias-free generative adversarial networks. In Adv. Neural Inform. Process. Syst.,\n2021.\nAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete\nXiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv\npreprint arXiv:2304.02643, 2023.\nChen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten\nKreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content\ncreation. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 300–309, 2023a.\nKai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.\nVision transformer for nerf-based view synthesis from a single input image. In IEEE Winter Conf.\nAppl. Comput. Vis., 2023b.\nMinghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su.\nOne-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization, 2023a.\nRuoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.\nZero-1-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11328, 2023b.\n11\n\n\nUnder review as a conference paper at ICLR 2024\nXiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang.\nSparseneus: Fast\ngeneralizable neural surface reconstruction from sparse views. 2022.\nTiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson.\nScalable 3d captioning with\npretrained models. arXiv preprint arXiv:2306.07279, 2023.\nLars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.\nOccupancy networks: Learning 3d reconstruction in function space. In IEEE Conf. Comput. Vis.\nPattern Recog., 2019.\nBen Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and\nRen Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Eur. Conf.\nComput. Vis., 2020.\nThomas M¨\nuller, Alex Evans, Christoph Schied, and Alexander Keller.\nInstant neural graphics\nprimitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, July\n2022.\ndoi: 10.1145/3528223.3530127.\nURL https://doi.org/10.1145/3528223.\n3530127.\nThu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan:\nUnsupervised learning of 3d representations from natural images. In Int. Conf. Comput. Vis.,\n2019.\nAlex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system\nfor generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.\nMichael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative\nneural feature fields. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.\nEvangelos Ntavelis, Aliaksandr Siarohin, Kyle Olszewski, Chaoyang Wang, Luc Van Gool, and\nSergey Tulyakov. Autodecoding latent 3d diffusion models. arXiv preprint arXiv:2307.05445,\n2023.\nJeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.\nDeepsdf: Learning continuous signed distance functions for shape representation. In IEEE Conf.\nComput. Vis. Pattern Recog., 2019.\nWilliam Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint\narXiv:2212.09748, 2022.\nBen Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d\ndiffusion. arXiv, 2022.\nGuocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-\nYing Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.\nMagic123: One image\nto high-quality 3d object generation using both 2d and 3d diffusion priors.\narXiv preprint\narXiv:2306.17843, 2023.\nAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal,\nGirish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual\nmodels from natural language supervision. In International conference on machine learning, pp.\n8748–8763. PMLR, 2021.\nRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨\norn Ommer. High-\nresolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern\nRecog., 2022.\nKatja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields\nfor 3d-aware image synthesis. In Adv. Neural Inform. Process. Syst., 2020.\nZifan Shi, Sida Peng, Yinghao Xu, Geiger Andreas, Yiyi Liao, and Yujun Shen. Deep generative\nmodels on 3d representations: A survey. arXiv preprint arXiv:2210.15663, 2022.\n12\n\n\nUnder review as a conference paper at ICLR 2024\nJ. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d\nneural field generation using triplane diffusion. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.\nVincent Sitzmann, Michael Zollh¨\nofer, and Gordon Wetzstein.\nScene representation networks:\nContinuous 3d-structure-aware neural scene representations. Advances in Neural Information\nProcessing Systems, 32, 2019.\nVincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.\nImplicit neural representations with periodic activation functions. Advances in neural information\nprocessing systems, 33:7462–7473, 2020.\nVincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field\nnetworks: Neural scene representations with single-evaluation rendering. Advances in Neural\nInformation Processing Systems, 34:19313–19325, 2021.\nIvan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training\nof 3d gans. In Adv. Neural Inform. Process. Syst., 2022.\nIvan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka,\nand Sergey Tulyakov.\n3d generation on imagenet.\nIn International Conference on Learning\nRepresentations, 2023. URL https://openreview.net/forum?id=U2WjB9xxZ9q.\nJiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv\npreprint arXiv:2010.02502, 2020a.\nYang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben\nPoole. Score-based generative modeling through stochastic differential equations. arXiv preprint\narXiv:2011.13456, 2020b.\nStanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(0-) image-\nconditioned 3d generative models from 2d data. arXiv preprint arXiv:2306.07881, 2023.\nAyush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan,\nChristoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Ad-\nvances in neural rendering. In Computer Graphics Forum, volume 41, pp. 703–735. Wiley Online\nLibrary, 2022.\nHaochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich.\nScore\njacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.\narXiv preprint\narXiv:2212.00774, 2022.\nQianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T\nBarron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-\nview image-based rendering. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.\nZhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.\nProlificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.\narXiv preprint arXiv:2305.16213, 2023.\nYinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via\nlearning structural and textural representations. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.\nYinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan\nYang, Yujun Shen, Hsin-Ying Lee, Bolei Zhou, et al.\nDiscoscene: Spatially disentangled\ngenerative radiance fields for controllable 3d-aware scene synthesis.\nIn IEEE Conf. Comput.\nVis. Pattern Recog., 2023.\nAlex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from\none or few images. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.\nXianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan,\nChenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of\nmulti-view images. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 9150–9161, 2023.\n13\n\n\nUnder review as a conference paper at ICLR 2024\nTable 4: Robustness on GSO dataset.\nLighting/Fov\nAppearance\nGeometry\nFID ↓\nCLIP ↑\nPSNR ↑\nSSIM ↑\nLPIPS ↓\nCD ↓\nOurs\n30.01\n0.928\n22.57\n0.845\n0.126\n0.0395\nFov10\n35.69\n0.912\n19.136\n0.820\n0.207\n0.0665\nFov30\n32.309\n0.921\n20.428\n0.839\n0.166\n0.0527\nFov70\n32.095\n0.921\n20.961\n0.860\n0.154\n0.0616\nFov90\n34.438\n0.912\n19.952\n0.855\n0.190\n0.0754\ncity\n33.31\n0.916\n21.19\n0.831\n0.142\n0.0437\nnight\n36.32\n0.907\n20.383\n0.829\n0.161\n0.0413\nsunrise\n33.264\n0.917\n21.080\n0.843\n0.140\n0.0423\nstudio\n36.32\n0.927\n21.383\n0.839\n0.141\n0.0428\nInput\nw. MvImageNet\nw.o. MvImageNet\nFigure 7: Qualitative comparison on w. and w.o. MvImageNet.\nA\nAPPENDIX\nA.1\nROBUSTNESS EVALUATION.\nWe evaluate our model with different FOV angles and lighting conditions to justify its robustness.\nSpecifically, while the MVImgNet datasets include diverse camera FOVs and lighting conditions,\nour model is mostly trained with 50◦-FOV and uniform lighting from the Objaverse dataset. We\nevaluate the robustness of our model (image-conditioned one) by testing images with other FOV\nangles and complex environment maps. As shown in Tab. 4, our model is sensitive to the FOV angles\nof the captured images, leading to lower quality with angles more deviated from the trained one. In\ngeneral, our model assumes an input image with a 50◦FOV, thus causing visible shape distortion\nin generated 3D shapes when the input FOV is different. However, it exhibits lower sensitivity to\nlighting variations, leading to similar quality across different lighting conditions. When the lighting\nis non-uniform, despite not physically matching the input, our model bakes the shading effects into\nthe NeRF appearance, yielding plausible renderings.\nA.2\nQUANTATIVE EVALUATION ON MVIMAGENET.\nMvImageNet contains a diverse set of real data, which helps to improve our generalization\ncapabilities for real data or out-of-domain data, as demonstrated in Fig 7.\nWe also perform\nquantative evaluation on the model with and without MvImageNet on the GSO dataset in Tab. 5. The\nreconstructed results in terms of appearance and geometry are similar to the previous results only\ntrained with Objaverse, indicating that MvImageNet improves generalization without compromising\nthe quality of reconstruction.\nA.3\nIMPLEMENTATION DETAILS.\nPlease see Tab. 6 for details.\n14\n\n\nUnder review as a conference paper at ICLR 2024\nTable 5: Ablation on MvImageNet.\n#Views\nAppearance\nGeometry\nFID ↓\nCLIP ↑\nPSNR ↑\nSSIM ↑\nLPIPS ↓\nCD ↓\nw. MvImageNet\n30.01\n0.928\n22.57\n0.845\n0.126\n0.0395\nw.o MvImageNet 27.761\n0.924\n21.851\n0.850\n0.128\n0.0378\nSmall\nLarge\nEncoder\nAtt Layers\n12\n12\nPatch size\n16\n8\nDecoder\nTriplane tokens\n323\n643\nChannels\n32\n32\nAtt layers\n12 (a +c)\n16 (a+c)\nRenderer\nToken upsample\n1\n2\nPatch size\n64\n128\nSteps\n48\n96\nDiffusion\nSteps\n1000\n1000\nLearn sigma\nFalse\nFalse\nPredict target\nx0\nx0\nSchedule\ncosine\ncosine\nTraininig\nLearning rate\n4e-4\n4e-4\nOptimizer\nAdamw\nAdamw\nWarm-up\n3000\n3000\nTable 6: Implementation details.\nA.4\nVIEW NUMBERS\nWe have compared the effects of using different numbers of views quantitatively in Tab. 3. Here,\nwe also present qualitative results in Fig. 8. When there is only one view, the predicted novel view\nis very blurry. However, when the view number increases to four, the results become much clearer.\nWhen using six views, the improvement compared to four views is not significant, consistent to\nthe metrics reported in Tab. 3, indicating saturation. Therefore, our network uses four views as the\ndefault configuration.\nA.5\nMORE COMPARISON.\nWe also include more qualitative comparison on single-view image reconstruction in Fig. 9.\n15\n\n\nUnder review as a conference paper at ICLR 2024\nInput\n#view 1\n#view 4\n#view 2\n#view 6\nFigure 8: Qualitative comparison on difference view numbers.\nShapE\nPoint-E\nOne-2345 Magic123\nOurs\nFigure 9: Qualitative comparison on single-image reconstruction.\n16","difficulty":"hard","domain":"Multi-Document QA","length":"short","question":"In the Phidias model, the loss function for reference-augmented multi-view diffusion is expressed as:\n\\[\nL = \\mathbb{E}{t,\\epsilon \\sim \\mathcal{N}(0,1)} \\left[ \\lVert \\epsilon - \\epsilon\\theta(x_t, t, c_{\\text{image}}, c_{\\text{ref}}) \\rVert^2 \\right]\n\\]\nwhere:\n\t•\t \\epsilon_\\theta  is the predicted noise at each timestep.\n\t•\t x_t  is the noisy image at timestep  t .\n\t•\t c_{\\text{image}}  is the conditioning on the input concept image.\n\t•\t c_{\\text{ref}}  is the conditioning on the 3D reference model (expressed as canonical coordinate maps, or CCMs).\nThe Meta-ControlNet in Phidias modifies the strength of the conditioning based on the alignment between the reference and the concept image.\nGiven this architecture, how does Meta-ControlNet influence the gradients during backpropagation, particularly in handling misaligned references during the training process, and why is this modulation essential to improving generalization in 3D generation?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=9c5f8d95-e6b6-5d14-890f-89b3d02a83c2&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}