Realistic Graphics Lab

Reparameterizing discontinuous integrands for differentiable rendering

In Transactions on Graphics (Proceedings of SIGGRAPH Asia 2019)

The solu­tion of in­verse ren­der­ing prob­lems us­ing gradi­ent-based op­tim­iz­a­tion re­quires es­tim­ates of pixel de­riv­at­ives with re­spect to ar­bit­rary scene para­met­ers. We fo­cus on the prob­lem of com­put­ing such de­riv­at­ives for para­met­ers that af­fect vis­ib­il­ity, such as the po­s­i­tion and shape of scene geo­metry (a, c) and light sources (b, d). Our ren­der­er re-para­met­er­izes in­teg­rals so that their gradi­ents can be es­tim­ated us­ing stand­ard Monte Carlo in­teg­ra­tion and auto­mat­ic dif­fer­en­ti­ation—even when vis­ib­il­ity changes would nor­mally make the in­teg­rands non-dif­fer­en­ti­able. Our tech­nique pro­duces high-qual­ity gradi­ents at low sample counts (64 spp in these ex­amples) for changes in both dir­ect and in­dir­ect vis­ib­il­ity, such as glossy re­flec­tions (a, b) and shad­ows (c, d).


Dif­fer­en­ti­able ren­der­ing has re­cently opened the door to a num­ber of chal­len­ging in­verse prob­lems in­volving photoreal­ist­ic im­ages, such as com­pu­ta­tion­al ma­ter­i­al design and scat­ter­ing-aware re­con­struc­tion of geo­metry and ma­ter­i­als from pho­to­graphs. Dif­fer­en­ti­able ren­der­ing al­gorithms strive to es­tim­ate par­tial de­riv­at­ives of pixels in a rendered im­age with re­spect to scene para­met­ers, which is dif­fi­cult be­cause vis­ib­il­ity changes are in­her­ently non-dif­fer­en­ti­able.

We pro­pose a new tech­nique for dif­fer­en­ti­at­ing path-traced im­ages with re­spect to scene para­met­ers that af­fect vis­ib­il­ity, in­clud­ing the po­s­i­tion of cam­er­as, light sources, and ver­tices in tri­angle meshes. Our al­gorithm com­putes the gradi­ents of il­lu­min­a­tion in­teg­rals by ap­ply­ing changes of vari­ables that re­move or strongly re­duce the de­pend­ence of the po­s­i­tion of dis­con­tinu­it­ies on dif­fer­en­ti­able scene para­met­ers. The un­der­ly­ing para­met­er­iz­a­tion is cre­ated on the fly for each in­teg­ral and en­ables ac­cur­ate gradi­ent es­tim­ates us­ing stand­ard Monte Carlo sampling in con­junc­tion with auto­mat­ic dif­fer­en­ti­ation. Im­port­antly, our ap­proach does not rely on sampling sil­hou­ette edges, which has been a bot­tle­neck in pre­vi­ous work and tends to pro­duce high-vari­ance gradi­ents when im­port­ant edges are found with in­suf­fi­cient prob­ab­il­ity in scenes with com­plex vis­ib­il­ity and high-res­ol­u­tion geo­metry. We show that our meth­od only re­quires a few samples to pro­duce gradi­ents with low bi­as and vari­ance for chal­len­ging cases such as glossy re­flec­tions and shad­ows. Fi­nally, we use our dif­fer­en­ti­able path tracer to re­con­struct the 3D geo­metry and ma­ter­i­als of sev­er­al real-world ob­jects from a set of ref­er­ence pho­to­graphs.


Text citation

Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. In Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38(6).

    author = {Guillaume Loubet and Nicolas Holzschuch and Wenzel Jakob},
    title = {Reparameterizing discontinuous integrands for differentiable rendering},
    journal = {Transactions on Graphics (Proceedings of SIGGRAPH Asia)},
    volume = {38},
    number = {6},
    year = {2019},
    month = nov,
    doi = {10.1145/3355089.3356510}