Path-space Motion Estimation and Decomposition for Robust Animation Filtering

In Computer Graphics Forum (Proceedings of Eurographics Symposium on Rendering 2015)

Start­ing from noisy, low-res­ol­u­tion frames gen­er­ated with a path tracer (red bor­ders), our meth­od im­proves qual­ity and re­duces com­pu­ta­tion­al cost by com­put­ing spa­tially and tem­por­ally up­sampled and de­noised frames (green and blue bor­ders) while prop­erly pre­serving view-de­pend­ent shad­ing ef­fects like the re­flec­tions in the pic­ture frame and on the ro­bot.


Ren­der­ings of an­im­a­tion se­quences with phys­ics-based Monte Carlo light trans­port sim­u­la­tions are ex­ceed­ingly costly to gen­er­ate frame-by-frame, yet much of this com­pu­ta­tion is highly re­dund­ant due to the strong co­her­ence in space, time and among samples. A prom­ising ap­proach pur­sued in pri­or work en­tails sub­sampling the se­quence in space, time, and num­ber of samples, fol­lowed by im­age-based spa­tio-tem­por­al up­sampling and de­nois­ing. 

These meth­ods can provide sig­ni­fic­ant per­form­ance gains, though ma­jor is­sues re­main: first, in a mul­tiple scat­ter­ing sim­u­la­tion, the fi­nal pixel col­or is the com­pos­ite of many dif­fer­ent light trans­port phe­nom­ena, and this con­flict­ing in­form­a­tion causes ar­ti­facts in im­age-based meth­ods. Secondly, mo­tion vec­tors are needed to es­tab­lish cor­res­pond­ence between the pixels in dif­fer­ent frames, but it is un­clear how to ob­tain them for most kinds of light paths (e.g. an ob­ject seen through a curved glass pan­el). 

To re­duce these am­bi­gu­ities, we pro­pose a gen­er­al de­com­pos­i­tion frame­work, where the fi­nal pixel col­or is sep­ar­ated in­to com­pon­ents cor­res­pond­ing to dis­joint sub­sets of the space of light paths. Each com­pon­ent is ac­com­pan­ied by mo­tion vec­tors and oth­er aux­il­i­ary fea­tures such as re­flect­ance and sur­face nor­mals. The mo­tion vec­tors of spec­u­lar paths are com­puted us­ing a tem­por­al ex­ten­sion of man­i­fold ex­plor­a­tion and the re­main­ing com­pon­ents use a spe­cial­ized vari­ant of op­tic­al flow. Our ex­per­i­ments show that this de­com­pos­i­tion leads to sig­ni­fic­ant im­prove­ments in three im­age-based ap­plic­a­tions: de­nois­ing, spa­tial up­sampling, and tem­por­al in­ter­pol­a­tion.



Text citation

Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, David Adler, Oliver Wang, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space Motion Estimation and Decomposition for Robust Animation Filtering. In Computer Graphics Forum (Proceedings of Eurographics Symposium on Rendering) 34(4). 131--142.

    author = {Wenzel Jakob},
    title = {Path-space Motion Estimation and Decomposition for Robust Animation Filtering},
    journal = {Computer Graphics Forum (Proceedings of Eurographics Symposium on Rendering)},
    volume = {34},
    number = {4},
    pages = {131--142},
    year = {2015},
    month = jun,
    doi = {10.1111/cgf.12685}