Netflix Explains and Develops FGS
A few weeks ago, we promised to get back to you on the subject of Film Grain Synthesis (FGS) which is a mandatory part of the AV1 codec. The promise was triggered by a video talk by Li-Heng Chen of Netflix and published by the Alliance for Open Media (AOM) which promotes and develops the AV1 and upcoming AV2 codecs.
Just as a background, and those ‘in the know’ can jump the next couple of paragraphs, FGS is a potentially very useful tool for improving the perceived quality of compressed video while keeping bit rates down. Film grain is considered an important part of the ‘look’ of content captured on film. However, because of the random nature of the grain, it is very difficult to compress efficiently. That means that if you want to maintain the ‘Creator’s Intent’ in film-based content, with an accurate representation of the grain, you need a very high bit rate.
The idea of FGS is that you analyze the grain in the source material and then remove it from the video. A description of the grain is passed as metadata through the codec and then the decoder adds back the grain. This allows very significant bandwidth savings, while, proponents claim, maintaining the look of the original. The Fraunhofer HHI has shown impressive demonstrations of its FGS technology at IBC and NAB. FGS is a mandatory part of the AV1 codec decoding specification, but is optional in the VVC codec. However, there are critics of the implementation of FGS in the AV1 codec, so, as Netflix is using it more and more, we were interested in their take on the subject.
The Talk – Why is FGS Hard?
Chen said that Netflix is supporting FGS in AV1 at scale now and he highlighted a Japanese Drama ‘The Hot Spot’ which has a lot of film grain. He also identified Department Q as having used FGS and Baahubali 2, an Indian film. (Note that if you watch the YouTube video, there were technical issues, so the ‘meat’ of the talk starts at 3 minutes 46 seconds).
The first topic of the full talk was the challenge in compressing grain – why is it so hard? One of the key reasons is that the pattern is completely random so two consecutive frames with the same image content would have different grain. Each frame, effectively, is introducing a random variable to the process. Chen showed a section of a frame from a Netflix movie compressed at around 5Mbps and highlighted how the compression artifacts were ‘quite disgusting’ (his words!). You can see DCP patterns, chroma patterns and ringing among other issues.
Chen explained the flow chart that we show above. The tool was invented in 2018. AV1 looks at the pattern of the grain (as finer or coarser) and the intensity, that is to say, the contrast within the grain pattern. These factors can be different even as you look around just a single frame. An autoregressive (AR) function is used to model the noise and Chen said that a 2D AR model can describe the grain shape ‘quite well’. The value of a pixel is calculated by looking at adjacent pixels. This is used to create a 64 x 64 template of the noise from the grain structure in the frame. The intensity is modelled by a scaling function that is based on the brightness of the pixel (as grain is not visible in very dark areas of the image).
Metadata Sent to the Decoder
The 64 x 64 template and the intensity function are sent to the decoder which uses a 32 x 32 sample of the template and the intensity data to render the new frame. A random seed is used to select where the 32 x 32 block is extracted from the template. There is a question of whether the random seed is random enough and whether a smaller block size would be better. In its roll-out, Netflix was able to reduce its bitrates by 30% (from grainy titles) with subjectively better image quality. The higher the resolution of the original, the more advantage was gained from FGS.
Chen showed a frame that had almost a three-fold decrease in bitrate, but which produced a higher quality image.
Note the 8KA has combined images from two of the slides to allow a clearer understanding of the quality issues, although the images are from YouTube capture. The original images are in this Netflix blog article
There is an additional benefit from the FGS process. Adding the grain structure masks the appearance of artifacts from the compression process. If you de-noise to eliminate the grain without FGS, the appearance is significantly downgraded but adding the grain back gets you closer to the original and with fewer artifacts. Chen emphasized that it is important to get the de-noising ‘just right’ and he said “Don’t be too greedy” in de-noising. You also need the right de-noiser but there are many options – although the better the de-noiser, the better the result.
The Seed is Important
Chen said that the seed is important. Some seeds give a good result but others give bad results so there is work to be done in this area still as some seeds can give ‘caterpillar’ artifacts. Chen said that Netflix had a technical blog article on this, but we couldn’t find it.1
Netflix produces a wide range of different types of content, so it uses adaptive streaming. Sometimes the denoising in FGS can produce ‘weird’ results in a particular resolution. If this is the case, Netflix turns off the denoising and FGS in all resolutions for the affected ‘chunk’ of content (typically a scene) as the resolution may change during a particular program, depending on network conditions. You need to know when the result is not good.
FGS is now used across the Netflix catalog from this year and the technology is supported in Google’s Chrome and Safari as well as some newer iPhones.
The Same Spec Can be Used in other Codecs
There is a new specification AFGS1 which can be used to send FGS parameters in Supplemental Enhancement Information (SEI) messages which means that the technique can be used in older codecs such as H.264 or H.265 (HEVC) if the decoder supports grain synthesis.
In AV2, which will be released shortly, there are improvements both to the FGS process and in the randomness. In AV2 you can also use a smaller grid of 16 x 16 blocks which reduces the chance of repeating patterns in the grain synthesis. You can also improve the references.
In questions, Chen was asked if the option on some recent digital still cameras of emulating film grain was based on the same technology. He said that it was different but that it was a good direction for the future. He was also asked how Netflix avoids degrading the image when de-noising, which can be destructive of detail. He said that Netflix uses a sophisticated de-noiser (which he would not identify) and said that you should just take a ‘thin layer of noise’ without breaking the edges in the image. In response to another question he said that he totally agreed that it would be good to take into account temporal aspects of the grain.
- Chen responded to our request for a reference and pointed us to this YouTube video. It has an interesting section on the decoding and encoding improvements in AV1. The discussion of FGS starts at around 12 minutes. The comments about seeds are at about 19 minutes and beyond. ↩︎
