RGL
Realistic Graphics Lab
EPFL Logo

Many-Worlds Inverse Rendering

Ziyi Zhang Nicolas Roussel Wenzel Jakob
In Transactions on Graphics (Proceedings of SIGGRAPH Asia 2025)

Left: (a) Pri­or dif­fer­en­ti­able ren­der­ing meth­ods com­pute how sur­face de­form­a­tions im­pact light trans­port through oc­clu­sion and shad­ing changes. The res­ult­ing gradi­ents drive loc­al geo­met­ric ad­just­ments. (b) Our meth­od in­stead con­siders adding hy­po­thet­ic­al sur­face patches any­where in 3D space. We sim­ul­tan­eously eval­u­ate many such patches as in­de­pend­ent, com­pet­ing ex­plan­a­tions of the in­put data. This ap­proach, termed many-worlds de­riv­at­ives, ex­tends gradi­ent com­pu­ta­tion from sur­faces in­to the sur­round­ing space. This com­bines the ro­bust­ness of volumes with the ef­fi­ciency of sur­face ren­der­ing: our meth­od does not re­quire an ini­tial mesh and can be star­ted from an empty scene, while avoid­ing the ex­pense of trans­mit­tance and mul­tiple scat­ter­ing com­pu­ta­tions. Right: An ex­ample re­con­struc­tion us­ing many-worlds de­riv­at­ives: a tri­angle mesh em­bed­ded in glass and ob­served through a mir­ror.

Resources

Paper (5.7 MB)
Supplemental material (749.5 KB)
Video (6.3 MB)
Publisher version
 Cite

Abstract

Dis­con­tinu­ous vis­ib­il­ity changes re­main a ma­jor bot­tle­neck when op­tim­iz­ing sur­faces with­in a phys­ic­ally based in­verse ren­der­er. Many pre­vi­ous works have pro­posed soph­ist­ic­ated al­gorithms and data struc­tures to sample vis­ib­il­ity sil­hou­ettes more ef­fi­ciently.

Our work presents an­oth­er solu­tion: in­stead of evolving a sur­face loc­ally, we ex­tend dif­fer­en­ti­ation to hy­po­thet­ic­al sur­face patches any­where in 3D space. We refer to this as a “many-worlds” rep­res­ent­a­tion be­cause it mod­els a su­per­pos­i­tion of in­de­pend­ent sur­face hy­po­theses that com­pete to ex­plain the ref­er­ence im­ages. These hy­po­theses do not in­ter­act through shad­ow­ing or scat­ter­ing, lead­ing to a new trans­port law that dis­tin­guishes our meth­od from pri­or work based on ex­po­nen­tial ran­dom me­dia.

The com­plete elim­in­a­tion of vis­ib­il­ity-re­lated dis­con­tinu­ity hand­ling by­passes the most com­plex and costly com­pon­ent of pri­or in­verse ren­der­ing meth­ods, while the ex­ten­ded de­riv­at­ive do­main pro­motes rap­id con­ver­gence. We demon­strate that the res­ult­ing Monte Carlo al­gorithm solves phys­ic­ally based in­verse prob­lems with both re­duced per-it­er­a­tion cost and few­er total it­er­a­tions.

Video

Figures

BibTeX
@article{10.1145/3767318,
    author = {Zhang, Ziyi and Roussel, Nicolas and Jakob, Wenzel},
    title = {Many-Worlds Inverse Rendering},
    year = {2025},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    issn = {0730-0301},
    url = {https://doi.org/10.1145/3767318},
    doi = {10.1145/3767318},
    journal = {ACM Trans. Graph.},
    month = sep,
    keywords = {differentiable rendering}
}