RGL
Realistic Graphics Lab
EPFL Logo

Large Steps in Inverse Rendering of Geometry

In ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2021)

(a) In­verse re­con­struc­tion of the Nefertiti bust from a spher­ic­al start­ing guess with 25 rendered views (1 shown). (b) Naïve ap­plic­a­tion of a dif­fer­en­ti­able ren­der­er pro­duces an un­us­able tangled mesh when gradi­ent steps pull on the sil­hou­ette without re­gard for dis­tor­tion or self-in­ter­sec­tions. (c) Reg­u­lar­iz­a­tion can al­le­vi­ate such prob­lems by mak­ing the op­tim­iz­a­tion aware of mesh qual­ity. On the flip­side, this pen­al­izes non-smooth parts of the geo­metry and causes un­sat­is­fact­ory con­ver­gence in gradi­ent-based op­tim­izers. While the fi­nal mesh un­deni­ably looks bet­ter, a closer in­spec­tion of the wire­frame ren­der­ing re­veals count­less self-in­ter­sec­tions. (d) Our meth­od ad­dresses both prob­lems and con­verges to a high-qual­ity mesh. (e) Com­bined with an iso­trop­ic remesh­ing step, our re­con­struc­tion cap­tures fine de­tails of the ref­er­ence (f). The hy­per-para­met­ers of each meth­od were op­tim­ized to ob­tain the best con­ver­gence at equal time. Self-in­ter­sec­tions are shown in red.

 

Abstract

In­verse re­con­struc­tion from im­ages is a cent­ral prob­lem in many sci­entif­ic and en­gin­eer­ing dis­cip­lines. Re­cent pro­gress on dif­fer­en­ti­able ren­der­ing has led to meth­ods that can ef­fi­ciently dif­fer­en­ti­ate the full pro­cess of im­age form­a­tion with re­spect to mil­lions of para­met­ers to solve such prob­lems via gradi­ent-based op­tim­iz­a­tion.

At the same time, the avail­ab­il­ity of cheap de­riv­at­ives does not ne­ces­sar­ily make an in­verse prob­lem easy to solve. Mesh-based rep­res­ent­a­tions re­main a par­tic­u­lar source of ir­rit­a­tion: an ad­verse gradi­ent step in­volving ver­tex po­s­i­tions could turn parts of the mesh in­side-out, in­tro­duce nu­mer­ous loc­al self-in­ter­sec­tions, or lead to in­ad­equate us­age of the ver­tex budget due to dis­tor­tion. These types of is­sues are of­ten ir­re­cov­er­able in the sense that sub­sequent op­tim­iz­a­tion steps will fur­ther ex­acer­bate them. In oth­er words, the op­tim­iz­a­tion lacks ro­bust­ness due to an ob­ject­ive func­tion with sub­stan­tial non-con­vex­ity.

Such ro­bust­ness is­sues are com­monly mit­ig­ated by im­pos­ing ad­di­tion­al reg­u­lar­iz­a­tion, typ­ic­ally in the form of Lapla­cian en­er­gies that quanti­fy and im­prove the smooth­ness of the cur­rent it­er­ate. However, reg­u­lar­iz­a­tion in­tro­duces its own set of prob­lems: solu­tions must now com­prom­ise between solv­ing the prob­lem and be­ing smooth. Fur­ther­more, gradi­ent steps in­volving a Lapla­cian en­ergy re­semble Jac­obi's it­er­at­ive meth­od for solv­ing lin­ear equa­tions that is known for its ex­cep­tion­ally slow con­ver­gence.

We pro­pose a simple and prac­tic­al al­tern­at­ive that casts dif­fer­en­ti­able ren­der­ing in­to the frame­work of pre­con­di­tioned gradi­ent des­cent. Our pre­con­di­tion­er bi­ases gradi­ent steps to­wards smooth solu­tions without re­quir­ing the fi­nal solu­tion to be smooth. In con­trast to Jac­obi-style it­er­a­tion, each gradi­ent step propag­ates in­form­a­tion among all vari­ables, en­abling con­ver­gence us­ing few­er and lar­ger steps.

Our meth­od is not re­stric­ted to meshes and can also ac­cel­er­ate the re­con­struc­tion of oth­er rep­res­ent­a­tions, where smooth solu­tions are gen­er­ally ex­pec­ted. We demon­strate its su­per­i­or per­form­ance in the con­text of geo­met­ric op­tim­iz­a­tion and tex­ture re­con­struc­tion.

Video

Video

Figures

Text citation

Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry. In ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 40(6).

BibTeX
@article{Nicolet2021Large,
    author = {Baptiste Nicolet and Alec Jacobson and Wenzel Jakob},
    title = {Large Steps in Inverse Rendering of Geometry},
    journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia)},
    volume = {40},
    number = {6},
    year = {2021},
    month = dec,
    doi = {10.1145/3478513.3480501}
}