RGL
EPFL Logo

Unbiased Inverse Volume Rendering with Differential Trackers

In Transactions on Graphics (Proceedings of SIGGRAPH 2022)

We demonstrate the high-quality reconstruction of volumetric scattering parameters from RGB images with known camera poses (left). This is enabled by our novel differential ratio tracking formulation, which yields unbiased, low-variance gradients of the radiative transfer equation that can be directly used for optimization.
Traditional free-flight sampling—e.g. by delta tracking—while effective at low-variance rendering, exhibits bias and high variance in gradient estimation with respect to medium density (top right), which negatively affects optimization. Gradient mean and variance values are shown for slice z = 64 of the 256 × 128 × 128 parameter space.
In the chart (bottom right), we report the improvements in reconstruction error for stochastic gradient descent with momentum (SGDm) as well as Adam. Using aggressive step size reduction, the Adam optimizer limits the impact of large gradient outliers, though our unbiased gradients lead to the lowest reconstruction error with either optimizer.

Abstract

Volumetric representations are popular in inverse rendering because they have a simple parameterization, are smoothly varying, and transparently handle topology changes. However, incorporating the full volumetric transport of light is costly and challenging, often leading practitioners to implement simplified models, such as purely emissive and absorbing volumes with “baked” lighting.

One such challenge is the efficient estimation of the gradients of the volume's appearance with respect to its scattering and absorption parameters. We show that the straightforward approach—differentiating a volumetric free-flight sampler—can lead to biased and high-variance gradients, hindering optimization. Instead, we propose using a new sampling strategy: differential ratio tracking, which is unbiased, yields low-variance gradients, and runs in linear time.

Differential ratio tracking combines ratio tracking and reservoir sampling to estimate gradients by sampling distances proportional to the unweighted transmittance rather than the usual extinction-weighted transmittance. In addition, we observe local minima when optimizing scattering parameters to reproduce dense volumes or surfaces. We show that these local minima can be overcome by bootstrapping the optimization from nonphysical emissive volumes that are easily optimized.


Results presented in the paper are best viewed through our interactive results viewer.

Code: our implementation of Differential Ratio Tracking, based on Mitsuba 3, is available under the BSD 3-clause license on GitHub.

Video

Figures

Text citation

Merlin Nimier-David, Thomas Müller, Alexander Keller, and Wenzel Jakob. 2022. Unbiased Inverse Volume Rendering with Differential Trackers. In Transactions on Graphics (Proceedings of SIGGRAPH) 41(4).

BibTeX
@article{nimierdavid2022unbiased,
	author = {Merlin Nimier-David and Thomas M\"uller and Alexander Keller and Wenzel Jakob},
	title = {Unbiased Inverse Volume Rendering with Differential Trackers},
	journal = {ACM Trans. Graph.},
	issue_date = {July 2022},
	volume = {41},
	number = {4},
	month = jul,
	year = {2022},
	pages = {44:1--44:20},
	articleno = {44},
	numpages = {20},
	url = {https://doi.org/10.1145/3528223.3530073},
	doi = {10.1145/3528223.3530073},
	publisher = {ACM},
	address = {New York, NY, USA},
	keywords = {differentiable rendering, inverse rendering, volumetric rendering, radiative backpropagation, importance sampling}
}