From Two Million Pixels to Six Hex Codes: How Palette Extraction Works

Open any photo in an image editor and check its unique color count. A modest 1200×900 JPEG routinely contains 30,000 to 100,000 distinct RGB values — and that's before counting the near-duplicates that JPEG compression and anti-aliasing create along every edge. A palette tool has to reduce that down to 6 or 8 swatches, and the obvious approach — tally how often each exact color appears, keep the most frequent ones — breaks immediately.

Why counting the top colors doesn't work

A cloudless sky doesn't render as one blue. It's a gradient, which means it's actually thousands of colors, each differing from its neighbor by a single unit of red, green, or blue. If you rank pixels by raw frequency, the top 8 "colors" in a landscape photo are usually eight near-identical shades of sky blue, because a gradient region simply has more total pixels than any single flat-colored object in the frame. The palette comes back looking like a blue swatch and seven variations of the same blue — technically the most common values, useless as a summary of the image.

What you actually want isn't the most frequent exact value. It's a small set of colors that each represent a meaningfully different region of the image — sky, foliage, skin tone, a red jacket — regardless of how many pixels each region happens to contain internally.

Median-cut: splitting color space, not counting pixels

Median-cut solves this by working in color space instead of frequency tables. Every pixel in the image is a point in a 3D cube where the axes are red, green, and blue (0–255 each). The algorithm starts with one box containing every pixel in the image, then repeats a simple step:

1. Find the box's longest axis — whichever of R, G, or B has the widest range of values among the pixels currently in that box.
2. Sort the box's pixels along that axis and split at the median, so each new box gets exactly half the pixels.
3. Repeat on each resulting box.

Each split doubles the number of boxes, so producing 8 swatches takes 3 splits, and 12 swatches (not a power of two) means splitting unevenly until 12 boxes exist. Once splitting stops, each box's final color is the average of every pixel it contains. The result is genuinely different from a frequency count: a box can be small in color-space distance but still get its own bucket if it's a tight, evenly-populated cluster, while a gradient sky — wide in color space but continuous — gets progressively subdivided into fewer, broader boxes instead of dominating every split. Spellkit's Color Palette tool uses exactly this method, offering 4, 6, 8, 10, or 12 swatches, and shows what share of the image each final box actually covers.

Why downsampling first barely changes the answer

Before running median-cut, it helps to shrink the image, and Color Palette downsamples every input to at most 200 pixels on its long side. The speed gain is large — a 4000×3000 photo is 12 million pixels; downsampled to 200×150 it's about 30,000, a 400x reduction, since shrinking both dimensions by 20x cuts total pixels by 20².

The result barely changes because median-cut never cared about the exact pixel count, only the proportions between regions. Downsampling is essentially averaging over small neighborhoods, so a region that was 40% of the image at full resolution is still roughly 40% at 200px — the median splits land in nearly the same places. What downsampling does change is noise: a handful of stray pixels (a single blown-out highlight, a compression artifact) that would otherwise get boxed off as their own tiny cluster get blended into their surroundings instead, which is a feature, not a loss of accuracy.

Dominant isn't the same as meaningful

This is where a tool's output and a human designer's instinct diverge. Median-cut ranks purely by pixel count, so a large gray overcast sky, a plain wall, or a wide stretch of pavement will out-rank a small red logo or a person's blue jacket every time, even though a designer looking at the same photo would call the red or blue the "important" color and barely register the sky. The algorithm has no concept of visual salience, subject matter, or brand relevance — it only knows how much area each color-cluster covers.

That's also why the percentage figure next to each swatch matters more than the swatch order. A 45% gray next to a 3% red tells you plainly that the gray is background and the red is the accent worth designing around — information the raw hex codes alone don't convey. If a specific small accent color is missing from the results entirely, it's usually because it fell under roughly 1% of the frame and got absorbed into a neighboring box during downsampling; cropping to just that region before re-running isolates it.