Unbiased Inverse Volume Rendering with Differential Trackers

In Transactions on Graphics (Proceedings of SIGGRAPH 2022)

We demon­strate the high-qual­ity re­con­struc­tion of volu­met­ric scat­ter­ing para­met­ers from RGB im­ages with known cam­era poses (left). This is en­abled by our nov­el dif­fer­en­tial ra­tio track­ing for­mu­la­tion, which yields un­biased, low-vari­ance gradi­ents of the ra­di­at­ive trans­fer equa­tion that can be dir­ectly used for op­tim­iz­a­tion.
Tra­di­tion­al free-flight sampling—e.g. by delta track­ing—while ef­fect­ive at low-vari­ance ren­der­ing, ex­hib­its bi­as and high vari­ance in gradi­ent es­tim­a­tion with re­spect to me­di­um dens­ity (top right), which neg­at­ively af­fects op­tim­iz­a­tion. Gradi­ent mean and vari­ance val­ues are shown for slice z = 64 of the 256 × 128 × 128 para­met­er space.
In the chart (bot­tom right), we re­port the im­prove­ments in re­con­struc­tion er­ror for stochast­ic gradi­ent des­cent with mo­mentum (SG­Dm) as well as Adam. Us­ing ag­gress­ive step size re­duc­tion, the Adam op­tim­izer lim­its the im­pact of large gradi­ent out­liers, though our un­biased gradi­ents lead to the low­est re­con­struc­tion er­ror with either op­tim­izer.


Volu­met­ric rep­res­ent­a­tions are pop­u­lar in in­verse ren­der­ing be­cause they have a simple para­met­er­iz­a­tion, are smoothly vary­ing, and trans­par­ently handle to­po­logy changes. However, in­cor­por­at­ing the full volu­met­ric trans­port of light is costly and chal­len­ging, of­ten lead­ing prac­ti­tion­ers to im­ple­ment sim­pli­fied mod­els, such as purely emissive and ab­sorb­ing volumes with “baked” light­ing.

One such chal­lenge is the ef­fi­cient es­tim­a­tion of the gradi­ents of the volume's ap­pear­ance with re­spect to its scat­ter­ing and ab­sorp­tion para­met­ers. We show that the straight­for­ward ap­proach—dif­fer­en­ti­at­ing a volu­met­ric free-flight sampler—can lead to biased and high-vari­ance gradi­ents, hinder­ing op­tim­iz­a­tion. In­stead, we pro­pose us­ing a new sampling strategy: dif­fer­en­tial ra­tio track­ing, which is un­biased, yields low-vari­ance gradi­ents, and runs in lin­ear time.

Dif­fer­en­tial ra­tio track­ing com­bines ra­tio track­ing and reser­voir sampling to es­tim­ate gradi­ents by sampling dis­tances pro­por­tion­al to the un­weighted trans­mit­tance rather than the usu­al ex­tinc­tion-weighted trans­mit­tance. In ad­di­tion, we ob­serve loc­al min­ima when op­tim­iz­ing scat­ter­ing para­met­ers to re­pro­duce dense volumes or sur­faces. We show that these loc­al min­ima can be over­come by boot­strap­ping the op­tim­iz­a­tion from non­phys­ic­al emissive volumes that are eas­ily op­tim­ized.

Res­ults presen­ted in the pa­per are best viewed through our in­ter­act­ive res­ults view­er.

Code: our im­ple­ment­a­tion of Dif­fer­en­tial Ra­tio Track­ing, based on Mit­suba 3, is avail­able un­der the BSD 3-clause li­cense on Git­Hub.



Text citation

Merlin Nimier-David, Thomas Müller, Alexander Keller, and Wenzel Jakob. 2022. Unbiased Inverse Volume Rendering with Differential Trackers. In Transactions on Graphics (Proceedings of SIGGRAPH) 41(4).

	author = {Merlin Nimier-David and Thomas M\"uller and Alexander Keller and Wenzel Jakob},
	title = {Unbiased Inverse Volume Rendering with Differential Trackers},
	journal = {ACM Trans. Graph.},
	issue_date = {July 2022},
	volume = {41},
	number = {4},
	month = jul,
	year = {2022},
	pages = {44:1--44:20},
	articleno = {44},
	numpages = {20},
	url = {https://doi.org/10.1145/3528223.3530073},
	doi = {10.1145/3528223.3530073},
	publisher = {ACM},
	address = {New York, NY, USA},
	keywords = {differentiable rendering, inverse rendering, volumetric rendering, radiative backpropagation, importance sampling}