CraGL: Computational Reality, Creativity and Graphics Lab

Differentiable Heightfield Path Tracing with Accelerated Discontinuities

Xiaochun Tong, Hsueh-Ti Derek Liu, Yotam Gingold, Alec Jacobson
ACM SIGGRAPH North America 2023 (Conference Papers).

Paper

Presentation

Left: The geometry of a sign and a small image with ambiguous lettering. Center: A house with the sign above the door. When rendered with morning illumination, the letters on the sign spell OPEN. Right: The same house rendered with afternoon illumination. The letters on the sign spell CLOSED. Our method enables real-time performance on inverse rendering tasks on heightfields. For instance, we use our renderer to optimize a heightfield geometry with texture (left) such that the “open” sign appears during the day and the “closed” sign appears in the afternoon.

Abstract

We investigate the problem of accelerating a physically-based differentiable renderer for heightfields based on path tracing with global illumination. On a heightfield with 1 million vertices (1024×1024 resolution), our differentiable renderer requires only 4 ms per sample per pixel when differentiating direct illumination, orders of magnitude faster than most existing general 3D mesh differentiable renderers. It is well-known that one can leverage spatial hierarchical data structures (e.g., the maximum mipmaps) to accelerate the forward pass of heightfield rendering. The key idea of our approach is to further utilize the hierarchy to speed up the backward pass—differentiable heightfield rendering. Specifically, we use the maximum mipmaps to accelerate the process of identifying scene discontinuities, which is crucial for obtaining accurate derivatives. Our renderer supports global illumination. we are able to optimize global effects, such as shadows, with respect to the geometry and the material parameters. Our differentiable renderer achieves real-time frame rates and unlocks interactive inverse rendering applications. We demonstrate the flexibility of our method with terrain optimization, geometric illusions, shadow optimization, and text-based shape generation.

Video Demonstration (46 seconds)

Download as:

BibTeX

(ACM Digital Library) 

@inproceedings{Tong:2023:DHP,
 author    = {Tong, Xiaochun and Liu, Hsueh-Ti Derek and Gingold, Yotam and Jacobson, Alec},
 title     = {Differentiable Heightfield Path Tracing with Accelerated Discontinuities},
 year      = {2023},
 booktitle = {SIGGRAPH 2023 Conference Proceedings},
 series    = {SIGGRAPH '23},
 keywords  = {differentiable rendering, inverse rendering, heightfield rendering, shading-based editing},
 doi       = {10.1145/3588432.3591530}
}