Skip to Main Content
June 18, 2024

Adobe Points the Way for Video Upscaling

Researchers at the University of Maryland, College Park and at Adobe Research have published an open access paper describing work they have done to generate a very high quality upscaling technology that is suitable for use with video content. 

We have previously written about the basics of upscaling of images (see here and here). However, dealing with each frame individually can create high frequency detail that varies from frame to frame causing flickering between adjacent frames when the content is displayed as video. Adobe published a paper about its GigaGAN static image upscaler last year. The approach was based on a generative adversarial network (GAN) technique for AI learning.  The researchers combined the GAN approach with training using billions of images to create the GigaGAN. We heard from Samsung earlier this year about how it also uses trained AI in its upscaling.

Avoiding Flicker

Adobe said that previous video targeted approaches have avoided the flickering problem using a range of techniques but these have tended to produce blurrier results. The new paper includes interesting examples comparing them and it’s worth a look at the results shown on the webpage. The group claims that its new VideoGigaGAN approach maintains the fine detail of single frame technologies, but also avoids the artifacts previously seen.

The basic concept of the approach is to use the scaling of the GigaGAN static upsampler to create a very high resolution image and then use a range of techniques to deal with potential artifacts. One of these is a recurrent flow-guided feature propogation model. Working out which parts of an image represent real movement in objects is difficult as adjacent frames may not be aligned at the pixel level. 

A second technique is to use anti-aliasing to minimize flickering. However, as this reduces the high frequency detail and can lead to blurriness, the group developed a way of directly shuttling the high frequency features back into the image at exactly the right place to minimize the flicker. The pdf of the group’s work addresses the details of this process.  

Limitations

In the paper, the researchers acknowledge that the system can get confused with feature propogation, which becomes inaccurate. It also struggles with small objects (such as small character text). The paper also does not comment on the time taken to process the images although the previous work on GigaGAN talked of seconds per image. Nor does it reference the level of processor power needed, so it is not, it seems, destined for use immediately in real-time applications such as in upscaling TVs. However, we can usually rely on Moore’s Law to solve some of the challenges of speeding processing up.

Nevertheless, the paper shows how much progress is being made in upscaling, which will be a significant part of helping the 8K ecosystem get over the ‘chicken and egg’ problem of limited 8K native resolution content.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x