b<>com’s VVC Testing With 4K and 8K Content
Last June, The 8K Monitor reported on b<> com’s 8K compression and UX testing. At that time, they had just purchased a Zaxtar 5 server for playout of uncompressed 8K content. We promised a follow-up on the HEVC and VVC 8K testing results, which we can now detail below.
The testing had three distinct goals.
- To evaluate the compression gains in using VVC over HEVC with 8K and confirm (or not) that they represent 40% to 50% improvements for the same visual quality,
- Determine the required bitrate for transparency, i.e. for there to be no visible differences before and after the encoding,
- Confirm that regular viewers can see the difference between 4K and 8K resolutions and measure that difference.
Charles Bonnineau, Ph. D. student at IRT b<>com, TDF, and INSA Rennes, has provided this fact-packed feedback looking into where 8K resolution can show a visible enhancement over 4K. The subjective and objective testing finds a visible difference between 8K and 4K content, but it is content-dependent. Specific camera movements at frame rates below 100fps can look worse in 8K as motion blur can be more apparent. b<> com’s initial results confirm the 41% bitrate reduction for 8K when using VVC vs. HEVC.
By Charles Bonnineau.
8K video resolution has attracted a lot of interest from the industry as a media format for immersive video applications enhancing the end user’s quality of experience. Nevertheless, 8K resolution brings many practical challenges increasing the end-to-end system latency and bandwidth requirement, which raise questions on its benefits in terms of quality enhancement. To answer these questions, we have conducted formal subjective experiments showing the benefits of 8K resolution over 4K relying on the two latest video coding standards HEVC and VVC.
For this test, the performance of the standards was assessed using their respective reference test model implementations that provide upper-bound performances with a limited level of optimization. The tests were performed using six video sequences with various spatial and temporal characteristics collected from two different sources: the Japanese organization ITE and Fraunhofer HHI. For each scene, the test points were obtained using additional fixed quantization parameter (QP) values. To cover a wide range of visual quality, we determined the highest bitrate value considering the transparency, i.e., the bitrate for which degradation starts to appear, as the highest bitrate point for each sequence. Also, the bitrate levels were carefully selected so that each bitrate Ri is approximately half of the next bitrate Ri+1 and each VVC bitrate level is close to the corresponding HEVC bitrate level. Each sequence was encoded and decoded at each bitrate level using the standard reference model. The decoded video was then stored on the Zaxel server for test purposes.
HEVC: HM-16.20, Random Access, GOP 16, Intra Period 48, Main10 profile.
VVC: VTM-11, Random Access, GOP 16, Intra Period 48, next profile.
Being able to store and play a large amount of 8K raw data was one of the main challenges of this test. This experiment was successfully carried out thanks to an efficient testing workflow set up by b<>com. The Zaxel server is the starting point of this pipeline. An initial conversion is made from the server’s 4 DisplayPort outputs to 4 HDMI 2.0 signals. Then, an Astrodesign device converts the 4 HDMI 2.0 inputs into a single HDMI 2.1 output connected to an 8K Sony TV.
The Double Stimulus Continuous Quality Scale (DSCQS) was selected as the testing methodology for this study. This testing method, included in the BT500 ITU Recommendation, is well-known among the community and ensures a fair subjective evaluation. After all the viewers participated in the test, a DMOS score representing the perceived quality was computed for each tested sequence. First, the rate-distortion curves for each scene are calculated to display the DMOS obtained for different selected bitrates. Those graphs also include the subjective quality assessed for both 4K and 8K uncompressed versions of the scenes (referred to respectively as “4K” and “ref” lines). This information allows us to evaluate the degree of relevancy of the 8K sequence and demonstrate that 8K offers a gain in visual quality compared to 4K with their confidence intervals.
The results demonstrate that the perceived quality between uncompressed 8K and 4K formats depends on the scene content. Thus, for the sequences JapaneseMaple, SteelPlant, BodeMuseum, and LayeredKimono, the visual quality between both resolutions is significantly different. For sequences with non-significant visible differences between 8K and 4K resolutions (Festival2 and OberbaumSpree), the motion in the scene can explain the 8K definition loss at 60fps. Indeed, the global motion in the Festival2 video sequence prevents perceiving the details. For the OberbaumSpree motion blur appears on the scene due to a continuous horizontal camera traveling. It shows that higher frame rates, e.g., 100/120fps, must be considered to fully benefit from the 8K resolution. Those curves also show that the bitrate required to reach the source quality using VVC ranges from 11 Mbps to 180 Mbps depending on the sequence.
In this test, we also quantify the gain offered by VVC over HEVC by using the widely used Bjontegaard delta method. This metric indicates the proportion of bitrate saved using VVC vs. HEVC at a similar objective or subjective quality level. The values for all sequences are reported in the Table below.
| Sequence | BD-BR (PSNR) | BD-BR (MS-SSIM) | BD-BR (VMAF) | BD-BR (DMOS) |
| LayeredKimono | -29.77% | -21.05% | -33.30% | -43.82% |
| BodeMuseum | -32.75% | -25.05% | -34.70% | -40.20% |
| OberbaumSpree | -32.07% | -27.00% | -33.41% | -54.30% |
| Festival2 | -36.40% | -33.36% | -28.24% | -32.66% |
| JapaneseMaples | -28.33% | -23.37% | -30.86% | -44.81% |
| SteelPlant | -28.30% | -24.40% | -27.57% | -34.11% |
| Average | -31.27% | -25.7% | -35.30% | -41.65% |
Table 1 DMOS BD-BR values
These results show that the average subjective gain offered by VVC over HEVC for 8K video is around 41%. At the same time, objective improvement ranges from approximately 26% to 35%, depending on the selected objective metric. The contribution of this work can be summarized as follows:
- VVC outperforms HEVC for 8K video with around 41% bitrate saving on average for similar visual quality.
- The required bandwidth to convey 8K video at high quality is content-dependent and may vary in the range of 11 Mbps to 180 Mbps with VVC.
- Quality enhancement brought by 8K resolution over 4K is content-dependent: high-quality acquisition sensors are essential, and a high frame rate (>60fps) is necessary for content containing significant motion.
