Realistic Graphics Lab

Large Steps in Inverse Rendering of Geometry

To appear in ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2021)

(a) In­verse re­con­struc­tion of the Nefertiti bust from a spher­ic­al start­ing guess with 25 rendered views (1 shown). (b) Naïve ap­plic­a­tion of a dif­fer­en­ti­able ren­der­er pro­duces an un­us­able tangled mesh when gradi­ent steps pull on the sil­hou­ette without re­gard for dis­tor­tion or self-in­ter­sec­tions. (c) Reg­u­lar­iz­a­tion can al­le­vi­ate such prob­lems by mak­ing the op­tim­iz­a­tion aware of mesh qual­ity. On the flip­side, this pen­al­izes non-smooth parts of the geo­metry and causes un­sat­is­fact­ory con­ver­gence in gradi­ent-based op­tim­izers. While the fi­nal mesh un­deni­ably looks bet­ter, a closer in­spec­tion of the wire­frame ren­der­ing re­veals count­less self-in­ter­sec­tions. (d) Our meth­od ad­dresses both prob­lems and con­verges to a high-qual­ity mesh. (e) Com­bined with an iso­trop­ic remesh­ing step, our re­con­struc­tion cap­tures fine de­tails of the ref­er­ence (f). The hy­per-para­met­ers of each meth­od were op­tim­ized to ob­tain the best con­ver­gence at equal time. Self-in­ter­sec­tions are shown in red.



In­verse re­con­struc­tion from im­ages is a cent­ral prob­lem in many sci­entif­ic and en­gin­eer­ing dis­cip­lines. Re­cent pro­gress on dif­fer­en­ti­able ren­der­ing has led to meth­ods that can ef­fi­ciently dif­fer­en­ti­ate the full pro­cess of im­age form­a­tion with re­spect to mil­lions of para­met­ers to solve such prob­lems via gradi­ent-based op­tim­iz­a­tion.

At the same time, the avail­ab­il­ity of cheap de­riv­at­ives does not ne­ces­sar­ily make an in­verse prob­lem easy to solve. Mesh-based rep­res­ent­a­tions re­main a par­tic­u­lar source of ir­rit­a­tion: an ad­verse gradi­ent step in­volving ver­tex po­s­i­tions could turn parts of the mesh in­side-out, in­tro­duce nu­mer­ous loc­al self-in­ter­sec­tions, or lead to in­ad­equate us­age of the ver­tex budget due to dis­tor­tion. These types of is­sues are of­ten ir­re­cov­er­able in the sense that sub­sequent op­tim­iz­a­tion steps will fur­ther ex­acer­bate them. In oth­er words, the op­tim­iz­a­tion lacks ro­bust­ness due to an ob­ject­ive func­tion with sub­stan­tial non-con­vex­ity.

Such ro­bust­ness is­sues are com­monly mit­ig­ated by im­pos­ing ad­di­tion­al reg­u­lar­iz­a­tion, typ­ic­ally in the form of Lapla­cian en­er­gies that quanti­fy and im­prove the smooth­ness of the cur­rent it­er­ate. However, reg­u­lar­iz­a­tion in­tro­duces its own set of prob­lems: solu­tions must now com­prom­ise between solv­ing the prob­lem and be­ing smooth. Fur­ther­more, gradi­ent steps in­volving a Lapla­cian en­ergy re­semble Jac­obi's it­er­at­ive meth­od for solv­ing lin­ear equa­tions that is known for its ex­cep­tion­ally slow con­ver­gence.

We pro­pose a simple and prac­tic­al al­tern­at­ive that casts dif­fer­en­ti­able ren­der­ing in­to the frame­work of pre­con­di­tioned gradi­ent des­cent. Our pre­con­di­tion­er bi­ases gradi­ent steps to­wards smooth solu­tions without re­quir­ing the fi­nal solu­tion to be smooth. In con­trast to Jac­obi-style it­er­a­tion, each gradi­ent step propag­ates in­form­a­tion among all vari­ables, en­abling con­ver­gence us­ing few­er and lar­ger steps.

Our meth­od is not re­stric­ted to meshes and can also ac­cel­er­ate the re­con­struc­tion of oth­er rep­res­ent­a­tions, where smooth solu­tions are gen­er­ally ex­pec­ted. We demon­strate its su­per­i­or per­form­ance in the con­text of geo­met­ric op­tim­iz­a­tion and tex­ture re­con­struc­tion.



Text citation

Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large Steps in Inverse Rendering of Geometry. In ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 40(6).

    author = {Baptiste Nicolet and Alec Jacobson and Wenzel Jakob},
    title = {Large Steps in Inverse Rendering of Geometry},
    journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia)},
    volume = {40},
    number = {6},
    year = {2021},
    month = dec,
    doi = {10.1145/3478513.3480501}