Mod­ern geo­met­ric re­con­struc­tion tech­niques achieve im­press­ive levels of ac­cur­acy in in­door en­vir­on­ments. However, such cap­tured data typ­ic­ally keeps light­ing and ma­ter­i­als en­tangled. It is then im­possible to ma­nip­u­late the res­ult­ing scenes in photoreal­ist­ic set­tings, such as aug­men­ted / mixed real­ity and ro­bot­ics sim­u­la­tion. Moreover, vari­ous im­per­fec­tions in the cap­tured data, such as miss­ing de­tailed geo­metry, cam­era mis­align­ment, un­even cov­er­age of ob­ser­va­tions, etc., pose chal­lenges for scene re­cov­ery. To ad­dress these chal­lenges, we present a ro­bust op­tim­iz­a­tion pipeline based on dif­fer­en­ti­able ren­der­ing to re­cov­er phys­ic­ally based ma­ter­i­als and il­lu­min­a­tion, lever­aging RGB and geo­metry cap­tures. We in­tro­duce a nov­el tex­ture-space sampling tech­nique and care­fully chosen in­duct­ive pri­ors to help guide re­con­struc­tion, avoid­ing low-qual­ity or im­plaus­ible loc­al min­ima. Our ap­proach en­ables ro­bust and high-res­ol­u­tion re­con­struc­tion of com­plex ma­ter­i­als and il­lu­min­a­tion in cap­tured in­door scenes. This en­ables a vari­ety of ap­plic­a­tions in­clud­ing nov­el view syn­thes­is, scene edit­ing, loc­al & glob­al re­light­ing, syn­thet­ic data aug­ment­a­tion, and oth­er photoreal­ist­ic ma­nip­u­la­tions.