CraGL: Computational Reality, Creativity and Graphics Lab

ColorfulCurves: Palette-Aware Lightness Control and Color Editing via Sparse Optimization

Cheng-Kang Ted Chao, Jason Klein, Jianchao Tan, Jose Echevarria, Yotam Gingold
ACM SIGGRAPH North America 2023 (Journal Papers). Also in ACM Transactions on Graphics (TOG) 42(4).

Paper

Presentation

Code

User Study Data

Three example images edited using ColorfulCurves. The first example shows palette-based luminance editing of a photograph of a waterfall over red cliffs. The user has placed constraints on the tone curves for some of the palettes to enhance constrast for some of the colors. The second example shows sparse optimization using image-space constraints. The photograph is of a city with sky, a statue of a man holding an umbrella floating in the air, and two buildings. A constraint has been placed on one of the buildings changing its color. The third example shows sparse optimization with mixed constraints. The photograph is of a green lake surrounding by mountains with people in canoes. A constraint has been placed on the water to turn it blue. A constraint has been placed on the trees to turn them red. Some constraints have been placed on the tone curves to enhance constrast for some of the colors. ColorfulCurves extracts a hue-chroma palette and builds palette-based tone curves to allow sparse, per-palette-color control of lightness over the image. Users place constraints on palette colors, tone curves, or directly on image pixels. ColorfulCurves optimizes the palette colors and lightness curves to satisfy the user’s constraints. Left: The user adds contrast to the rocks with S-shaped curve constraints on the red and brown colors. Center: The user places an image-space constraint on the building to make it dark brown. ColorfulCurves optimizes for the sparsest satisfying change to the palette. Right: The user places a mix of image-space, palette, and curve constraints. Photos courtesy of (left to right) Jeremy Bishop, Nastya Dulhiier, and Pietro De Grandi.

Abstract

Color editing in images often consists of two main tasks: changing hue and saturation, and editing lightness or tone curves. State-of-the-art palette-based recoloring approaches entangle these two tasks. A user’s only lightness control is changing the lightness of individual palette colors. This is inferior to state-of-the-art commercial software, where lightness editing is based on flexible tone curves that remap lightness. However, tone curves are only provided globally or per color channel (e.g., RGB). They are unrelated to the image content. Neither tone curves nor palette-based approaches support direct image-space edits—changing a specific pixel to a desired hue, saturation, and lightness. ColorfulCurves solves both of these problems by uniting palette-based and tone curve editing. In ColorfulCurves, users directly edit palette colors’ hue and saturation, per-palette tone curves, or image pixels (hue, saturation, and lightness). ColorfulCurves solves an L_2,1 optimization problem in real-time to find a sparse edit that satisfies all user constraints. Our expert study found overwhelming support for ColorfulCurves over experts’ preferred tools.

Fast Forward (20 seconds)

Download as:

Video Demonstration (4:30 minutes)

Download as:

BibTeX

(ACM Digital Library entry) 

@article{Chao:2023:CC,
 author    = {Chao, Cheng-Kang Ted and Klein, Jason and Tan, Jianchao and Echevarria, Jose and Gingold, Yotam},
 title     = {Colorful{C}urves: Palette-Aware Lightness Control and Color Editing via Sparse Optimization},
 journal   = {ACM Transactions on Graphics (TOG)},
 volume    = {42},
 number    = {4},
 year      = {2023},
 month     = jul,
 url       = {https://doi.org/10.1145/3592405},
 doi       = {10.1145/3592405},
 articleno = {98},
 numpages  = {12},
 keywords  = {palette-based image editing, color, optimization, lightness, tone curves, usability}
}