ReCap: Better Gaussian Relighting with Cross-Environment Captures

Abstract

Accurate 3D objects relighting in diverse unseen environments is crucial for realistic virtual object placement. Due to the albedo-lighting ambiguity, existing methods often fall short in producing faithful relights. Without proper constraints, observed training views can be explained by numerous combinations of lighting and material attributes, lacking physical correspondence with the actual environment maps used for relighting. In this work, we present ReCap, treating cross-environment captures as multi-task target to provide the missing supervision that cuts through the entanglement. Specifically, ReCap jointly optimizes multiple lighting representations that share a common set of material attributes. This naturally harmonizes a coherent set of lighting representations around the mutual material attributes, exploiting commonalities and differences across varied object appearances. Such coherence enables physically sound lighting reconstruction and robust material estimation — both essential for accurate relighting.

Motivation

Existing methods enabled relighting of Gaussians using explicit shading functions and learnable lighting representations, often in the form of environment maps. However, these methods often fail to produce faithful relights with only single-environment captures as inputs.

Due to the albedo-lighting ambiguity, where changes in surface albedo are indistinguishable from changes in lighting intensity, the learned environments are often observed to be tinted with object colors, shifted in tone, scaled in intensity or filled with noise. Without proper constraints, these maps act as sinks for unmodeled residual terms during optimization, lacking physical correspondence with the actual environment maps used for relighting.

The same object position can display a range of pixel colors depending on viewing direction and lighting. Unlike existing methods, we explicitly model light dependent appearances with multiple learnable environment maps.

Learned environment maps

Inspired by photometric appearance modeling, we propose ReCap to leverage object captures across unknown lighting conditions, modeling light-dependent appearances with multiple environment maps that share a common Gaussian model. Conceptually, this resembles multi-task learning, where the learned environment maps act as task heads querying a shared material representation for varied object appearances.

Method Overview

The proposed ReCap training framework. Compared to original 3DGS, each Gaussian is augmented with 3 extra material attributes. Given k sets of object appearances from unknown lighting conditions as input, k learnable environment maps are instantiated. Gaussian color is pre-computed according to the shading function in the world space based on environment queries and material properties. 2D images are rasterized with standard Gaussian splatting and used for loss computation.

References

GShader: Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. In CVPR, 2024.

GS-IR: Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. In CVPR, 2024

3DGS-DR: Keyang Ye, Qiming Hou, and Kun Zhou. 3d gaussian splatting with deferred reflection. In ACM SIGGRAPH, 2024.

R3DG: Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing. In ECCV, 2024

BibTeX