CraGL: Computational Reality, Creativity and Graphics Lab

LoCoPalettes: Local Control for Palette-based Image Editing

Cheng-Kang Ted Chao, Jason Klein, Jianchao Tan, Jose Echevarria, Yotam Gingold
Eurographics Symposium on Rendering. Also in Computer Graphics Forum (CGF), to appear.

Paper

Presentation

Code

Supplementary Material

An example image edited using LoCoPalettes. On the left we see a photograph labeled Input. The photograph is of a dog standing on green grass with pumpkins and trees visible in the background. A point on a green leaf in the background has been selected and its color constrained to change from brown to green. An arrow labeled Optimization connects the photograph to a modified version labeled Global Edit. In this photograph, the leaf in the background and the grass have both become brown. A six-color palette swatch below the image shows that the green color in the palette has become brown. An arrow labeled Optimization activates hierachy connects this center photograph to another modified photograph on the right. The photograph on the right is labeled Localized Edits. We see a point in the grass has been selected and its color constrained to a pale green. The grass has changed from the brown in the middle photograph to pale green. An arrow from the constraint points to two more copies of the palette, showing a palette changing from one set of colors to another. A small greyscale image next to these palettes is black where the grass is and white elsewhere. LoCoPalettes augments palette-based image editing with image-space constraints and semantic hierarchies. Left: The user places a spatial constraint on the input image to make the leaves brown. LoCoPalettes optimizes for a palette with respect to users’ image-space constraints, shown as small white circles. Center: The optimization updates the global palette. Right: The user places a second spatial constraint on the grass. If constraints cannot be satisfied with a single global palette, LoCoPalettes semantically segments the image and activates a palette hierarchy H to achieve the color constraints using local palettes. The grass constraint’s semantic segmentation mask is shown in the lower-right.

Abstract

Palette-based image editing takes advantage of the fact that color palettes are intuitive abstractions of images. They allow users to make global edits to an image by adjusting a small set of colors. Many algorithms have been proposed to compute color palettes and corresponding mixing weights. However, in many cases, especially in complex scenes, a single global palette may not adequately represent all potential objects of interest. Edits made using a single palette cannot be localized to specific semantic regions. We introduce an adaptive solution to the usability problem based on optimizing RGB palette colors to achieve arbitrary image-space constraints and automatically splitting the image into semantic sub-regions with more representative local palettes when the constraints cannot be satisfied. Our algorithm automatically decomposes a given image into a semantic hierarchy of soft segments. Difficult-to-achieve edits become straightforward with our method. Our results show the flexibility, control, and generality of our method.

Video Demonstration (2:30 minutes)

Download as:

BibTeX

(Publisher page) 

@article{Chao:2023:CC,
 author    = {Chao, Cheng-Kang Ted and Klein, Jason and Tan, Jianchao and Echevarria, Jose and Gingold, Yotam},
 title     = {{LoCoPalettes}: Local Control for Palette-based Image Editing},
 journal   = {Computer Graphics Forum (CGF)},
 note      = {Special issue for Eurographics Symposium on Rendering (EGSR)},
 volume    = {42},
 number    = {4},
 year      = {2023},
 month     = jun,
 keywords  = {palette-based image editing, color, local, optimization, usability},
 doi       = {10.1111/cgf.14892}
}