Film Grain Processing Can Boost Video Quality
While the technical enthusiasts among us tend to think that in terms of resolutions, color, dynamic range and refresh, more is always better, those in the content creation often have a different point of view. A couple of years ago, Ben Schwarz wrote an article that clarified this difference and I especially liked this quote from the article:
“I don’t want my images to be THAT realistic. I make movies, not nature documentaries.”
So, content creators looking to evoke a particular mood or period will exploit the low frame rates, sometimes restricted color, optical distortions from anamorphic lenses or vignetting. If they are using film stock, then the choice of film and the way it is exposed and processed will all be taken into account. One of the defining characteristics of a particular film and development process is film grain.
So What is Film Grain?
Film grains are the silver halide crystals or dye clouds in film and are very different from the standardized grid layout of typical digital camera sensors. The pattern formed by the grain creates the picture, so you can think of grain as ‘analog pixels’.
Broadly speaking for most films, the grain size is proportional to the sensitivity of the film. That is to say, the finer the grain, the less sensitive the film. The sensitivity of the film will also affect other aspects of the capture process, such as the type of lens used, the aperture of the lens and the shutter speed. Those factors will also influence the look of the content.
Film grain and its look can also be influenced by the chemicals and development process and those (like me) that used to play around with different film stocks and developers when processing their own film will manipulate those things to get the look that they want. (And if I remember rightly from some 25 years ago, my favorites for black and white images were Ilford FP4 (ISO 125) and HP-5 with ID-11 developer and Pan-F (ISO 50) with Acutol when I wanted more detail). In those analog only days, limited ISO changes could also be made by manipulating the development process.
The big difference between digital sensors and film grain is that the grain is ‘stochastic’ which is to say it is random in position and size, although within a particular range for the film/development. It’s not neatly arranged in the ‘rows and columns’ of digital sensors (but means that it rarely runs into the kind of moiré issues that digital sensors sometimes hit). The random nature of the grain can make it difficult to characterize and describe it. Transmitting images with grain poses particular challenges in the digital domain. However, the grain is not entirely random – if it was, it would be noise.
Further, the structure of the grain can significantly affect the apparent resolution of the image. The contrast at the edges of the grain, if high, can give the impression of more sharpness, even where the grain size means that the image is not truly sharp (which is why I used Acutol for monochrome photography). So it is really important to include the look of film grain when showing film-captured content on digital devices.
Codecs and Grain
The detail of film grain is, effectively, very high frequency data that can easily be lost in the compression and decompression processes. At the recent IBC show, the Fraunhofer HHI demonstrated its technology for dealing with grain when using the VVC codec and they showed that without special technology, to transmit the grain as detail and accurately send a FullHD image would need 60 Mbps of bandwidth. On the other hand a regularly compressed image that still looked good, but without the grain structure, could be transmitted in just 2.5Mbps
To get around this huge bandwidth hurdle, two steps are needed. First, the image is analyzed to characterize the grain structure in the content (in a paper presented at IBC, the Fraunhofer details grain size, intensity, distribution and temporal variation as key parameters). That characterization is transmitted to the decoder as a small amount of metadata. During the decompression process, the metadata is used to synthesize the effect of the grain on the final image. In the Fraunhofer demo at IBC, this got the bandwidth back from 60 Mbps to 2.5Mbps, but visually, this viewer couldn’t see the difference in the grain look between the 2.5Mbps version and the 60Mbps content.
VVC and AV1
The AV1 codec mandates support of this kind of film grain process, but for VVC the feature is an option. In VVC, the analysis and synthesis methods are not defined, just the format of the data. One of the intriguing points that the Fraunhofer makes in its paper is that a distributor could add metadata to create grain effects even when they were not in the original, and as we have seen, that can impact the apparent sharpness and resolution of the image (fake news, anyone?).
Because the analysis and synthesis processes are not standardized, different methods can be used and Wavelet Beam of Germany also told us at IBC that it has such technology. As the Fraunhofer identifies in its paper, the first step in the grain analysis is to detect and eliminate noise in the signl and this is an area where Wavelet Beam claims to have very good technology.
An Unexpected Bonus….
An intriguing bonus of film grain simulation (FGS) is that when applied to a highly compressed video stream that is showing visible compression artifacts, it reduces the visibility of those effects. That does make us wonder if broadcasters that have control of the decompression side (i.e. those that control the STBs as well) might use the technique to improve the quality of any older film-based content in the future, while restricting the bandwidth needed!
Archival Use
Where high resolution is being used to build digital archives, 8K scanning and capture can help to record the original grain structure where the original content was of relatively low resolution. In this case, archivists would probably not want to merely store metadata, but capture the actual grain image. Image quality maven, Florian Friedrich presented a webinar that talked about this topic on YouTube.
