De-Bayering Confuses Things When Pixel Counting
We talked in the first part of this occasional series about the advantages of downsampling video from higher captured resolutions to lower delivery formats to avoid aliasing artifacts. In the article we mentioned the complications of de-Bayering and so we thought it would be useful to summarize what that means.
The most widely accepted model of human vision is that based on the idea that the eye sees red, green and blue (although signals from the photoreceptors in the eye are based on red-green difference, blue-yellow difference and brightness). However, camera sensors, based on CMOS semiconductor technology, do not respond directly to color, but to brightness. So, to generate color, a filter is put in front of the sensor – in the same way that a color filter array (CFA) is added to an LCD to create color images.
Sensors
The sensing wells (pixels) in the CMOS sensors are arranged in a grid and using optics, the image is focused onto that grid with the light passing through the CFA. The different colored areas of the CFA are arranged in a pattern and the most common is the ‘Bayer’ pattern. (There are others, but Bayer is the most frequently used)
Bayer Pattern CFA Image: Colin M.L. Burnett Â
At each of the locations of the grid, the data is only recorded for the particular color in that part of the CFA. You can see immediately that there are twice as many pixels for green as there are for blue and red. That matches the sensor response more closely to the human retina which is also more sensitive to green. That makes sense in evolutionary terms – in nature there are many more green objects than red or blue.Â
The effect of this architecture is that the sensor only has data for a quarter of the pixels in red and blue, and for half in green. To get around this. sensor designers interpolate, i.e. calculate an intermediate value for the data that is missing, using values from the adjacent pixels. That calculation could be simple, but in the best sensors, sophisticated algorithms are used to create the missing values. Some of these are proprietary to particular camera and sensor makers. This process is known as de-bayering or it can be known as demosaicing.
 This process of de-bayering is usually carried out in the camera when the image is to be output in a standardised RGB format, such as JPEG. However, if the image is downloaded as a RAW file (here, RAW actually just means ‘unprocessed’, like raw vegetables, rather than being a specific standardized format) it will often be in a format before the de-Bayering process. This allows software to be used to perform alternative algorithms for the process. A RAW file with non de-Bayered data will normally include metadata to indicate the type of CFA used.
(There is a nice viewer on the DXO.com website that shows an image before and after demosaicing and another with good and bad performance. The author uses this software to process his digital camera images from RAW files.)
Resolution & Pixel Counting
This process of demosaicing can have a significant impact on the resolution of the final image because of the type of processing involved.Â
This is one of the reasons that those that are chasing the highest levels of image quality look to other measurements rather than just counting the photosites on a sensor. For example, see this article from 2021 when Chris spoke to Pawel Achtel about camera sensor resolutions. In the article, there was an explanation of the different way that Arri and Blackmagic Design use their CFAs and demosaicing. Achtel’s tests suggested that the resolution of the Arri 4K x 3K sensor was higher than that of the Ursa 12K x 6K camera.
