AO Media Says ‘The Future of Innovation is Open’ (Part II)
Last week we reported on the first part of the AOMedia webinar on its AV1 and AV2 codecs. This is the second part, which was a panel discussion.
The panellists were:
- Debargha Mukherjee – Principal Engineer (Google)
- Anush Moorthy – Engineering Leader – Video Algorithms & Media Systems (Netflix)
- Adrian Grange – Senior Technical Program Manager (Google)
- Hassene Tmar – Technical Program Manager (Meta)
The panel introduced themselves before the questions started.
Grange responded to a question about the key milestones in AV1 adoption over the last decade. The big challenge, he said, was to get everything started from nothing, including process development. It was fairly straightforward for most of the development time, but there was a big ‘last gasp’ effort to get everything finalized, debugged and sorted out. The release of the specification in 2018 was a big moment for all the teams involved.
Dav1d Was Critical
The next key point was the development of the dav1d decoder which was pivotal in achieving success. It took a while for encoders to mature, but there was a reasonable encoder early in the release phase. After that, market adoption really depended on the roll-out of hardware, which always takes some time. (hardware companies really have to wait until they are confident that a specification is stable before committing – editor). AV1 turned out to be good for software decoding, with not much more complexity than earlier software decoders.
Tmar commented on the development of the SVT-AV1 encoder. At the outset, lessons were learned from other successful codecs such as AVC and it was recognized that widespread availability of hardware and software solutions would be really helpful in encouraging adoption. The first use case was Video on Demand (VoD), followed by live encoding and then real-time communications (RTC) and images. The group was very keen to add value without adding more complexity than the minimum. Simply adding complexity is not a good approach, Tmar said.
Allowing Roll-outs at Scale
The aim all along was to meet the needs to allow large companies to make roll-outs at very large scale, which is what has happened, Tmar explained. ‘It was a bit of a dream’ he said, but the biggest thing was to keep the decoding in mind while designing the encoding. Often there is a gap between those designing encoders and decoders, but that ‘gap’ is quite critical. The small gap for AV1 really made a difference when there was still a gap before hardware became available.
Responding to a question about HDR, Moorthy was happy to respond saying “The team has already done the work!”. AV1 includes support for HDR10+ and he referenced a tech blog on the subject, although he clarified that not all the Netflix catalogue currently supports it. Work continues on optimizing the encoding for HDR10+ content.
Mukherjee said that work started just 2.5 years before the finalization, but work on new codecs had already been done on VP10 by Google, Daala at Mozilla and on Thor from Cisco so the first step for AV1 development was to pick the best points from each. AV2 was very different and took longer to get to the same stage. During AV1 development, the groups were ‘making it up as we went along’ in terms of processes, but these are clearer during AV2 development. The better processes have helped in getting the best results.
Grange emphasized that there were a lot of VP10 tools ‘in the pipeline’ for AV1 and while the processes were similar, there were better details in AV2, such as the numbering of documents etc. Tests were also being developed alongside the encoders which allowed verification very quickly. There were no core code bugs, although there were some around different use cases with higher level syntax. There was less formal emphasis on minimizing complexity in AV1 compared to AV2.
AV2 was also clearly divided into a research phase run ‘by the engineers’ and then a second specification development phase. In AV1, the two processes were ‘mangled together’, Grange said. The two phase process was about 80% successful.
AV1 in Live & RTC
A question was raised about the performance targets for live and real-time communication (RTC) use cases. Tmar said that in the development of SVT-AV1, it was planned for way for the codec to be multi-threaded without losing the VoD performance. He pressured the team to achieve this, although it has made code maintenance harder. It allows someone to get good performance without using tiles or slices, which would be used traditionally for multi-threading.
Tmar defines the ‘live’ use case as having video that needs to be encoded with fixed latency, which could be one to two seconds. RTC means something faster and the target is to catch up with H.264 in that space, although it is difficult in lower resolutions, where RTC often is. In higher definitions, the compression gains can be retained which may be better than H.264.
AV2
The webinar then moved on to a discussion of AV2. The panelists were asked to nominate their favorite new capability of AV2.
Mukherjee said if he had to pick just one, he would choose one from the encoding side and that is of screen graphics encoding. These have seen a really big improvement in moving from AV1.
Multi-stream was added as a second new feature before Lemieux could stop him!
Moorthy said that he agreed that the ‘alpha video’ support (i.e. computer/screen graphics) really helps the codec. He said that because it is a standard feature in the profile ‘decoders just work’.
Multi-streams are a ‘favorite new trick’.
Grange highlighted the support of multi-streams as his “favorite new trick” in AV2. The codec can send multiple sub-streams, each of which can be decoded as a separate video. An example he gave was of a soccer match with a main screen and smaller sub-screens showing other games or matches or news. The application can be told how to render the different streams with e.g. ‘composition rules’ in metadata.
Tmar’s said that reference picture re-sampling is his preferred feature, although it was also in AV1. He doesn’t believe he has seen a good product that uses that feature, yet, although he is optmistic one will appear soon. The feature allows you to use different resolution images as references allowing changing resolutions in the bitstream without having to insert a key frame. It has been challenging in AV1 to get adoption, but it should be more widely adopted in AV2 implementations.
There was a question about whether upcoming versions of STV/AVx (AV1 or AV2) could have automatic film grain detection built-in? Tmar said that this has been a feature request, but that he was looking for help from those that create the grain to do this. He emphasized that ‘It’s an open community’ but such a feature is ‘a lot of work’, but he is hopeful that it will happen. Lemieux pointed out that even non-members were welcome to provide code for encoding or decoding.
Another question was about the addition of content provenance data within the codecs. Lemieux suggested that C2PA is probably the most widely adopted so far and asked if the topic had come up? Grange said that this data is really a separate layer in the data stream, although he does feel it should be considered within AOMedia. Again Lemieux encouraged participation by those involved in provenance technology.
What’s the Future?
Lemieux asked if AV2 was the end of the road for ‘traditional’ codecs based on DCTs? Grange responded by looking in his ‘crystal ball’. The idea of prediction, transform and DCT has been around for forty years or so and many have predicted the end of these codecs for ‘maybe 38.5 years’. Many approaches have been tried including fractals and wavelets, complete transforms etc, but still the codecs are DCT-based. Getting more and more out of the same approach with DCTs is getting harder (as MPEG has also found), so it’s a fair question. However, he thinks DCT is ‘not dead yet’. Machine learning (ML) is interesting and the group is looking at whether a DCT-based core stream with ML enhancements may be useful.
ML moves the whole process closer to a software solution, Grange said. DCT needs hardware with dedicated ASICs but with ML there is ‘a light at the end of the tunnel’ that could be that ML is so important to so many use cases that it will be incorporated in some form in all consumer devices. ML currently doesn’t work fast enough for video, he added, and combining CPUs and GPUs with ML components is difficult because of ’round trip’ times between the processors. To make ML important for video, there needs to be more development.
Ultimately, Grange believes that AOpen is likely to go to ML for ‘AV3’ or the next generation of codecs, but he would like to see a return to basics to look at other possible solutions rather than simply assuming it would be ML. ML seems to be the best solution today, but although ML has worked well on images, it is less clear that it is good for video. A lot depends on the kind of time scale for the next generation of codec.
Better Analysis is Helpful
If you could do a better job of analyzing incoming videos and formulating an optimum strategy for each video, that would be helpful, Grange thinks. A combination of current coding tools with a boost from ML may be an optimum, although nobody really knows.
Mukherjee sees ML as probably the way to go but he agrees that the number of operations you need to support 4K 60P or 8K 30P video, for example, very quickly surpasses the capacity of the largest LLMs today. LLMs are typically in the hundreds of tokens per second, but video uses a lot of pixel data. However, Mukherjee sees a need for innovation in the ‘lightweight models’ space in combination with conventional models. He expects that the next phase is likely to be decoding some part of the video using an NPU or GPU on a device, before using traditional hardware. Most of imaging ML has used big models, but video needs light models.
Tmar agrees that adding lightweight models to AV2 would be ‘A very hard to beat combination’. Even with AV1, you can see some ‘pristine’ 1080P at less than 1 Mbps. He has seen a screen demo, using the latest screen content tools at less than 10Kbps that looked ‘awesome’. There is still a need for high resolution and frame rate, but the combination of light MML tools with conventional technology quickly becomes ‘unbeatable’.
Metrics are Important
The question that concerns Tmar is ‘How are we going to measure it?’. AOpen really needs help with quality metrics and he referenced work by the Google team on the Universal Video Quality Model (UVQ) that can be leveraged.
Changes in the network ecosystem can influence codec requirements. With an infinitely fast or cheap network, there may be less or no need for better codecs, Grange said, in the case of video ‘You’ve still got to store it and encode it’. Video coding will remain important unless all three of these issues are solved. New use case such as VR could take huge amounts of bandwidth. With Generative AI, he asked, is video just a text stream of metadata?
Traditionally with deteriorating video or audio streams, you reduce the quality of video first until it stops, then turn to audio until that stops and you finally go to a text chat window. With multiple audio streams in immersive audio, there can be a big demand for bandwidth. There are many potential ways to deliver moving images rather than with video.
AOMedia has done a lot of work on metadata for immersive audio, but audio is now competing with video for bandwidth.
The media landscape is changing and, for example, a lot of people now watch video with audio and with subtitles. Moorthy says he often listens with a low level of audio to avoid disturbing sleeping children, but using the subtitles for some dialogue. He sees value in the conventional technologies but also in trying to add to that vast experience with some aspects of ML. He explained that it has always been the aim in the past to retain as much as possible of the original signal/pixel, mathematically based. Perhaps in the generative AI time, the measurement paradigm might have to change. If fidelity is less important than similarity, then the metrics change.
Moorthy also sees potential developments in use cases which may add need new features beyond simply compression efficiency. This may mean a need for new or better codecs.
The Role of Academia?
Lemieux asked if there was scope for different ways at looking at video encoding and that might benefit from input from academia. Tmar said that looking wide is useful, but individuals always have the point of view of their companies, so projects such as the definitions of aspects such as quality can come from academia.
Grange thinks ‘we are at a crossroads’ about which way to go. He looks at the work in Google Research on understanding images and wonders if their approaches could be applied to video. Is a frame just a collection of pixels or do you try to understand the objects in the scene to help your compression?
Conclusion
The comment that for AV1 “there was a big ‘last gasp’ effort to get everything finalized, debugged and sorted out” may also ring true for AV2, which was due to be released at the end of 2025, but had not been seen at the time of the publication of this article. AO Media had not responded to a request for clarification on the release date, but we would expect that there is currently frantic activity going on to get it launched.
(There was a ‘release candidate’ in the AV1 subreddit dated 5th Jan 2026)
