Recursive Control Variates for Inverse Rendering

In Transactions on Graphics (Proceedings of SIGGRAPH 2023)

We im­prove phys­ic­ally based dif­fer­en­ti­able ren­der­ing by in­tro­du­cing a re­curs­ive con­trol vari­ate in­to the op­tim­iz­a­tion loop. Left: a baseline meth­od [Vi­cini et al. 2021] re­peatedly renders noisy im­ages to com­pute gradi­ents. The noise leads to poor con­ver­gence. Right: ap­plied on top of the baseline, our re­curs­ive con­trol vari­ate re­duces the noise us­ing in­form­a­tion from sim­il­ar ren­der­ings in pri­or op­tim­iz­a­tion steps. This leads to a re­con­struc­tion that is much closer to the ref­er­ence im­age. The plots show the evol­u­tion of the loss func­tion and ren­der­ing vari­ance as the op­tim­iz­a­tion pro­gresses.


We present a meth­od for re­du­cing er­rors—vari­ance and bi­as—in phys­ic­ally based dif­fer­en­ti­able ren­der­ing (PBDR). Typ­ic­al ap­plic­a­tions of PBDR re­peatedly render a scene as part of an op­tim­iz­a­tion loop in­volving gradi­ent des­cent. The ac­tu­al change in­tro­duced by each gradi­ent des­cent step is of­ten re­l­at­ively small, caus­ing a sig­ni­fic­ant de­gree of re­dund­ancy in this com­pu­ta­tion. We ex­ploit this re­dund­ancy by for­mu­lat­ing a gradi­ent es­tim­at­or that em­ploys a re­curs­ive con­trol vari­ate, which lever­ages in­form­a­tion from pre­vi­ous op­tim­iz­a­tion steps. The con­trol vari­ate re­duces vari­ance in gradi­ents, and, per­haps more im­port­antly, al­le­vi­ates is­sues that arise from dif­fer­en­ti­at­ing loss func­tions with re­spect to noisy in­puts, a com­mon cause of drift to bad loc­al min­ima or di­ver­gent op­tim­iz­a­tions. We ex­per­i­ment­ally eval­u­ate our ap­proach on a vari­ety of path-traced scenes con­tain­ing sur­face and volumes and ob­serve that prim­al ren­der­ing ef­fi­ciency im­proves by a factor of up to 10.



Text citation

Baptiste Nicolet, Fabrice Rousselle, Jan Novák, Alexander Keller, Wenzel Jakob, and Thomas Müller. 2023. Recursive Control Variates for Inverse Rendering. In Transactions on Graphics (Proceedings of SIGGRAPH) 42(4).

    author = {Baptiste Nicolet and Fabrice Rousselle and Jan Novák and Alexander Keller and Wenzel Jakob and Thomas Müller},
    title = {Recursive Control Variates for Inverse Rendering},
    journal = {Transactions on Graphics (Proceedings of SIGGRAPH)},
    volume = {42},
    number = {4},
    year = {2023},
    month = aug,
    doi = {10.1145/3592139}