I’m writing some C code that does alpha compositing using integer math (fixed point), and I’m coming across the problem that 0xff isn’t quite 1.0. This is particularly problematic in the alpha channel. For example (0xff * 0xff) >> 8 equals 0xfe. So if you have a white pixel with alpha 0xff and RGB 0xff, and you try to premultiply your alpha channel into the color channels, you end up with 0xfe in the RGB channels, so you no longer really have white.

Here are some example functions that do alpha premultiplication and then alpha-compositing with premultiplied alpha, both of which have some variant of this problem. These are for little endian RGBA/BGRA, let’s ignore byte order issues for now:

unsigned premultiply_alpha(unsigned color)
{
        unsigned alpha = ((color>>24) & 0xff);
        unsigned rb = color & 0xff00ffU;
        unsigned g  = color & 0x00ff00U;

        rb = (rb * alpha) >> 8;
        g  = (g  * alpha) >> 8;

        return (color & 0xff000000) | (rb & 0xff00ff) | (g & 0x00ff00);
}

unsigned rgb_blend(unsigned background, unsigned color)
{
        unsigned alpha = color>>24;
        unsigned rb = background & 0xff00ffU;
        unsigned  g = background & 0x00ff00U;
        rb = (color & 0xff00ff) + ((0xff - alpha) * rb >> 8);
        g  = (color & 0x00ff00) + ((0xff - alpha) * g  >> 8);

        unsigned dest_alpha = background>>24;
        dest_alpha = alpha + dest_alpha - ((alpha*dest_alpha) >> 8);
        dest_alpha = dest_alpha < 255 ? dest_alpha : 255;

        return  (dest_alpha << 24) | (rb & 0xff00ff) | (g & 0x00ff00);
}

The first function has the exact issue I described in prose above, and the second one has a similar problem in the expression (0xff - alpha), which can correctly express zero, but not 1.0.

How do people usually handle this without making a round trip through floating point? I’m sure people have already studied and solved this issue. Thanks.