Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction

In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. In particular, our technique enables high-quality scene reconstruction from Large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. Our approach introduces a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortion, and demonstrates state-of-the-art performance on both synthetic and real-world datasets.

Conventional approaches require undistorting the image into perspective views compatible with 3DGS rasterization. As the field of view increases, pixel stretching becomes progressively severe, significantly compromising the quality of the reconstruction. In contrast, our cubemap resampling strategy maintains a consistent pixel density across the entire field of view. This approach, combined with our hybrid distortion field, makes use of the peripheral regions (the annular area outside the blue box) without severe distortion or pixel stretching. Moreover, our method can handle fields of view all the way up to 180°, as demonstrated by the green box, allowing for comprehensive and accurate reconstructions.