awesome-3D-editing

Open-source Toolboxes and Foundation Models

Methods	Task	Github
threestudio	T23D Generation
DreamGaussian	T23D Generation

Table of Contents

• Preliminary: milestones for generation and editing
  • Classical representation
  • Classical text-3d generation
  • Single-view reconstruction
• 3D general editing
  • 3D local editing
  • 3D (global) editing
  • Part-based editing
  • Inpainting
  • Sketch-based editing
  • Reference-image based editing
• Appearance-specific editing
  • Stylization
  • Re-texture
  • Re-light
• Deformation
  • Cage
  • Control point
• Avatar-related editing
• Scene editing
• 4D editing

Paper list (Latest Update: 2023/10/26)

Preliminary: milestones for generation and editing

Classical representation

Title	arXiv	WebSite	Github	Pub.
3D Gaussian Splatting for Real-Time Radiance Field Rendering				SIGGRAPH 2023
Instant Neural Graphics Primitives				SIGGRAPH 2022

Classical text-3d generation

Title	arXiv	WebSite	Github	Pub.
ATT3D: Amortized Text-to-3D Object Synthesis			-	ICCV 2023
FantASIA3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation				ICCV 2023
Zero-1-to-3: Zero-shot One Image to 3D Object				ICCV 2023
Magic3D: High-Resolution Text-to-3D Content Creation			-	CVPR 2023
Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model		-		ArXiv, 2023, 10
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image				ArXiv, 2023, 09
MVDream: Multi-view Diffusion for 3D Generation				ArXiv, 2023, 08
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization				ArXiv, 2023, 06
Shap·E: Generating Conditional 3D Implicit Functions		-		ArXiv, 2023, 05
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures		-		ArXiv, 2022, 11
DreamFusion: Text-to-3D using 2D Diffusion				ArXiv, 2022, 09
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation				ArXiv
Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models			-	3DV 2024

Single-view reconstruction

Title	arXiv	WebSite	Github	Pub.
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior				ICCV 2023
Zero-1-to-3: Zero-shot One Image to 3D Object				ICCV 2023
Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model		-		ArXiv, 2023, 10
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image				ArXiv, 2023, 09
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors				ArXiv, 2023, 06
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation				ArXiv

3D general editing

3D local editing

Title	arXiv	WebSite	Github	Pub.
FocalDreamer: Text-driven 3D Editing via Focal-fusion Assembly				ArXiv, 2023, 08
DreamEditor: Text-Driven 3D Scene Editing with Neural Fields			-	SIGGRAPH ASIA 2023
OBJect 3DIT: Language-guided 3D-aware Image Editing			-	NeurIPS 2023
Vox-E: Text-guided Voxel Editing of 3D Objects				ICCV 2023
Locally Stylized Neural Radiance Fields		-	-	ICCV 2023
Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields				ICCV 2023
Local 3D Editing via 3D Distillation of CLIP Knowledge		-	-	CVPR 2023
RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models		-	-	CVPR 2023
SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field				CVPR 2023
SKED: Sketch-guided Text-based 3D Editing		-	-	CVPR 2023
Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision			-	CVPR 2023
Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts				ArXiv
Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates		-	-	ArXiv
ED-NeRF: Efficient Text-Guided Editing of 3D Scene using Latent Space NeRF		-	-	ArXiv

3D (global) editing

Title	arXiv	WebSite	Github	Pub.
NeRF-Art: Text-Driven Neural Radiance Fields Stylization				TVCG 2023
3D-aware Blending with Generative NeRFs				ICCV 2023
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions				ICCV 2023
Vox-E: Text-guided Voxel Editing of 3D Objects				ICCV 2023
3D Semantic Subspace Traverser: Empowering 3D Generative Model with Shape Editing Capability		-		ICCV 2023
DreamBooth3D: Subject-Driven Text-to-3D Generation			-	ICCV 2023
Blending-NeRF: Text-Driven Localized Editing in Neural Radiance Fields			-	ICCV 2023
Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis			-	ICCV 2023
Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model			-	CVPR 2023
Frequency-Modulated Point Cloud Rendering with Easy Editing		-		CVPR 2023
IT3D: Improved Text-to-3D Generation with Explicit View Synthesis		-		ArXiv, 2023, 09
Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields		-	-	ArXiv, 2023, 06
Watch Your Steps: Local Image and Scene Editing by Text Instructions			-	ArXiv

Part-based editing

Title	arXiv	WebSite	Github	Pub.
PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervision				CVPR 2023
FocalDreamer: Text-driven 3D Editing via Focal-fusion Assembly				ArXiv, 2023, 08
VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization			-	ArXiv

Inpainting

Title	arXiv	WebSite	Github	Pub.
OBJect 3DIT: Language-guided 3D-aware Image Editing			-	NeurIPS 2023
Reference-guided Controllable Inpainting of Neural Radiance Fields			-	ICCV 2023
Removing Objects From Neural Radiance Fields				CVPR 2023
SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields				CVPR 2023
InpaintNeRF360: Text-Guided 3D Inpainting on Unbounded Neural Radiance Fields		-	-	ArXiv, 2023, 05
OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields		-		ArXiv, 2023, 05
NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors		-	-	ArXiv, 2022, 06

Sketch-based editing

Title	arXiv	WebSite	Github	Pub.
FantASIA3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation				ICCV 2023
Sketch2Mesh: Reconstructing and editing 3D shapes from sketches		-		ICCV 2021
SKED: Sketch-guided Text-based 3D Editing		-	-	CVPR 2023
Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures		-		ArXiv, 2022, 11

Reference-image based editing

Title	arXiv	WebSite	Github	Pub.
Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields				ICCV 2023
SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field				CVPR 2023
Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates		-	-	ArXiv

Appearance-specific editing

Stylization

Title	arXiv	WebSite	Github	Pub.
NeRF-Art: Text-Driven Neural Radiance Fields Stylization				TVCG 2023
TeSTNeRF: Text-Driven 3D Style Transfer via Cross-Modal Learning		-	-	IJCAI 2023
X-Mesh:Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance				ICCV 2023
Locally Stylized Neural Radiance Fields		-	-	ICCV 2023
Transforming Radiance Field with Lipschitz Network for Photorealistic 3D Scene Stylization		-	-	CVPR 2023
PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields				CVPR 2023
StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields				CVPR 2023
Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization				CVPR 2023
MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field		-	-	ArXiv
Text-driven Editing of 3D Scenes without Retraining				ArXiv

Re-texture

Title	arXiv	WebSite	Github	Pub.
ICE-NeRF: Interactive Color Editing of NeRFs via Decomposition-Aware Weight Optimization			-	ICCV 2023
Texture Generation on 3D Meshes with Point-UV Diffusion				ICCV 2023
X-Mesh:Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance				ICCV 2023
Text2Tex: Text-driven Texture Synthesis via Diffusion Models				ICCV 2023
NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing				ECCV 2022
PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields				CVPR 2023
StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields				CVPR 2023
Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization				CVPR 2023
TensoIR: Tensorial Inverse Rendering				CVPR 2023
Text-guided High-definition Consistency Texture Model		-	-	ArXiv, 2023, 05
Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion			-	ArXiv, 2023, 04
ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context			-	ArXiv
Text-driven Editing of 3D Scenes without Retraining				ArXiv
Dyn-E: Local Appearance Editing of Dynamic Neural Radiance Fields			-	ArXiv
TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images			-	ArXiv
RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes				ACM MM 2023
RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes				ACM MM 2023

Re-light

Title	arXiv	WebSite	Github	Pub.
ENVIDR: Implicit Differentiable Renderer with Neural Environment Lighting				ICCV 2023
ReNeRF: Relightable Neural Radiance Fields with Nearfield Lighting		-	-	ICCV 2023
IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis				ICCV 2023
Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes		-	-	CVPR 2023
TensoIR: Tensorial Inverse Rendering				CVPR 2023

Deformation

Title	arXiv	WebSite	Github	Pub.
Real-time Locally Injective Volumetric Deformation		-		TOG 2021
Variational Harmonic Maps for Space Deformation		-	-	TOG 2009
GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents			-	SIGGRAPH 2023
TextDeformer: Geometry Manipulation using Text Guidance				SIGGRAPH 2023
Neural Shape Deformation Priors				NeurIPS 2022
ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators		-		ICCV 2021
Deforming Radiance Fields with Cages				ECCV 2022
NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing				ECCV 2022
NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds				CVPR 2023
NeRF-Editing: Geometry Editing of Neural Radiance Fields		-		CVPR 2022
Neural Cages for Detail-Preserving 3D Deformations		-		CVPR 2020
SPIDR: SDF-based Neural Point Fields for Illumination and Deformation				ArXiv, 2022, 10
VolTeMorph: Realtime, Controllable and Generalisable Animation of Volumetric Representations		-	-	ArXiv, 2022, 08
MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field				ArXiv

Cage

Title	arXiv	WebSite	Github	Pub.
Neural Cages for Detail-Preserving 3D Deformations		-		CVPR 2020

Control point

Title	arXiv	WebSite	Github	Pub.
Real-time Locally Injective Volumetric Deformation		-		TOG 2021
Variational Harmonic Maps for Space Deformation		-	-	TOG 2009
ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators		-		ICCV 2021
EditableNeRF: Editing Topologically Varying Neural Radiance Fields by Key Points			-	CVPR 2023
NeRF-Editing: Geometry Editing of Neural Radiance Fields		-		CVPR 2022

Avatar-related editing

Title	arXiv	WebSite	Github	Pub.
AvatarStudio: Text-driven Editing of 3D Dynamic Human Head Avatars			-	SIGGRAPH ASIA 2023
Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization			-	SIGGRAPH 2023
CLIPFace: Text-guided Editing of Textured 3D Morphable Models				SIGGRAPH 2023
DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance			-	SIGGRAPH 2023
AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars				SIGGRAPH 2022
FDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models		-		NeurIPS 2023
DeformToon3D: Deformable 3D Toonification from Neural Radiance Fields				ICCV 2023
Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images		-	-	ICCV 2023
FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields			-	ICCV 2023
CLIP-Actor: Text-Driven Recommendation and Stylization for Animating Human Meshes				ECCV 2022
Learning Locally Editable Virtual Humans				CVPR 2023
MEGANE: Morphable Eyeglass and Avatar Network			-	CVPR 2023
UV Volumes for Real-time Rendering of Editable Free-view Human Performance				CVPR 2023
HumanGen: Generating Human Radiance Fields with Explicit Priors		-	-	CVPR 2023
TECA: Text-Guided Generation and Editing of Compositional 3D Avatars			-	ArXiv, 2023, 09
MagicAvatar: Multimodal Avatar Generation and Animation				ArXiv, 2023, 08
HeadSculpt: Crafting 3D Head Avatars with Text			-	ArXiv, 2023, 06
Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D Diffusion-based Editor			-	ArXiv, 2023, 05
Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion			-	ArXiv, 2022, 12
LatentSwap3D: Semantic Edits on 3D Image GANs				ArXiv
Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field		-		AAAI 2023
TADA! Text to Animatable Digital Avatars				3DV 2024

Scene editing

Title	arXiv	WebSite	Github	Pub.
IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis				ICCV 2023
Diffusion-based Generation, Optimization, and Planning in 3D Scenes			-	CVPR 2023
I2-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFs				CVPR 2023
Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes		-	-	CVPR 2023
SIEDOB: Semantic Image Editing by Disentangling Object and Background		-		CVPR 2023
Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details		-	-	CVPR 2023
NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing			-	ArXiv 2022
Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes				ArXiv

4D editing

Title	arXiv	WebSite	Github	Pub.
NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos		-		NeurIPS 2022
Text-guided High-definition Consistency Texture Model		-	-	ArXiv, 2023, 05
Dyn-E: Local Appearance Editing of Dynamic Neural Radiance Fields			-	ArXiv
Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image		-	-	ACM MM 2023