4by4 Inc. Upscaling Solves the Chicken & Egg Problem
There is a perennial ‘chicken and egg’ challenge for new display resolutions. In the early days of the technology, there is relatively little pre-rendered content to show at the native resolution. It takes time and money to render content at every different resolution, especially when the quality goes up so content creators and distributors are reluctant to support new formats, until there are enough display devices out in the market. On the other hand, consumers can be reluctant to buy new displays unless there is a lot of content already. Even after many years of HD production, standard definition is a significant share of even streamed and satellite content – and both of those distribution systems have relatively high levels of capacity.
Upscaling is a Key Technology
One of the key technologies that has allowed 8K displays to develop despite this challenge is upscaling; that is to say, taking lower resolution content and translating it to the new higher format. As we wrote in some articles last year (Upscaling Reviewed (Part 1) And 8K Upscaling Reviewed Part 2), there are simple ways of doing this, but in recent years, the availability of machine learning and AI have transformed the quality of the upscaling that can be performed to a remarkable degree. This is particularly true for the scaling built into the best 8K TVs which has really improved in the last couple of years.
The technology to perform this process has often been developed by TV set makers, but has also been enabled by specialist technology providers and we spoke to one of those, 4by4 Inc. which is based in Korea. 4by4 has developed PIXELL, an image quality enhancement solution that is based on AI-based deep learning.
Simple Scaling Doesn’t Cut it
Simply scaling the pixel resolutions doesn’t always meet the desire for a more impactful image, as we heard in the recent panel session at NAB and run by the Digital Cinema Society (DCS). Better contrast, dynamic range and color can also give the impression of higher resolution. The effect has been understood for many years and was even a factor in the way that black and white photography was developed*.
In order to get a real boost to image quality, then, the upscaling should also take into account the color performance and contrast and that is what the PIXELL technology does.
Math Alone is Not Enough
Of course, it’s not enough to simply ‘apply the math’ – and 4by4 undertakes continuous enhancement of its models using feedback from verified video quality experts. As always in the area of video quality, review by ‘golden eye’ experts is essential to get a good result. The Pixell technology is intended to enhance degraded and distorted video quality that is the result of the type of camera, shooting environment, lighting etc. It aims to provide an image that is closer to that seen with the naked eye.
The company has a number of submitted and granted patents related to upscaling from 2013 onwards covering AI-based methods and video enhancement based on sharpness, color and noise reduction. 4by4 believes that its technology has measurably better results than some other technologies based on metrics including the SSIM metric for objective measurement of video quality.
The company recently announced an upgrade to its service that provides access to high quality (4K and above) stock footage.
Multi-faceted improvements
The firm has a number of different processes to enhance image quality. It can, for example, process the colors in an image to optimize saturation and contrast using a convolutional neural network (CNN) for multi-layer processing.
Noise is often a challenge when processing video and the firm has a multi-frame (3,5,7) process to enhance reduce noise, flicker and jitter in a video by calculating the frames at one time. Once noise has been identified, it can be removed in a process known as Denoise Prior Image Restoration (DPIR). The firm also uses Recurrent-Generative Back-Projection Network (RBPN) techniques in combination with DPIR to minimize noise. Before such techniques were developed, TV makers often had to make trade-offs when sharpening video as less sophisticated techniques can cause an increase in noise. (Interestingly, in different parts of the world, there would be different perceptions. Viewers in Asia often preferred sharper images, despite the noise, while in Europe, for example, a softer image was fine as long as noise was suppressed. Some TV makers would set up their sets with different balances for different regions – Editor).
An AI-based approach allows sharpness optimization without some of the disadvantages of earlier mathematical approaches.
The different optimization models can be applied separately or together for the best result.
Technology is Used in Different Ways
4by4 Inc exploits these technologies in a number of ways. As well as providing stock footage, it can provide post-production services for content creators that want to boost the quality of their output. A recent project was to take a popular Korean programme, Sungkyunkwan Scandal, that was filmed 13 years ago and to create an 8K master from the original version.
* the use of ‘high acuity’ developers emphasized the contrast on the edges of the grains of silver and led to images that often looked ‘finer grained’ than genuinely fine grain film.
4by4 Inc. is an associate member of the 8K Association
