Machine learning-based intra-prediction methods for video coding have gained profound attention in recent years. However, most of such approaches use complex non-linear networks that increase the coding complexity significantly. In this paper, a data-driven recursive intra-prediction (DRIP) method is proposed which applies a linear network at the sub-block level for intra-prediction. Briefly, a given block is partitioned into sub-blocks, and by multiplying the reconstructed or predicted samples present at the top and left of a sub-block with a weight matrix (learned offline) its prediction is obtained. The sub-blocks are predicted recursively, using the same weight matrix, until the prediction of the block is complete. For further coding improvement, multiple such matrices are used resulting in multiple prediction modes for a block. The index of the chosen mode is transmitted to the decoder in the bit-stream. An example implementation of DRIP with 32 modes and 4 × 4 sub-blocks on top of the VVC Test Model (VTM version 23.6) is presented in this paper. It achieves an average Bjøntegaard-delta rate of 1.02% (luma channel) against the anchor using the all-intra− (AI) picture configuration.

With the emergence of the Versatile Video Coding standard (VVC), novel coding tools like matrix-based intra prediction and low-frequency non-separable transforms have been developed based on data-driven optimization methods. Motivated by the growing compression efficiency of learned nonlinear transforms in image coding, we incorporate a neural-network-based coding tool into the transform coding stage of VVC. First, a nonlinear update of the transform coefficients is applied which aims at decreasing the expected bitrate cost. Then, on the decoder side, a filter is applied before the synthesis transform to improve the reconstruction quality. These networks are jointly trained and have been added to the rate-distortion optimized quantization process. The integration of these approaches into the VTM software leads to bitrate savings between 0.94% and 2.12% in terms of the Bjøntegaard-Delta rate. We furthermore conducted multiple experiments on reducing the memory complexity of the networks by exploiting structural similarities of the intra prediction modes and block symmetries. As a result, we demonstrate that the number of distinct networks can be reduced without significantly diminishing the coding gain of the tool. With memory usage reduced by 90%, we achieve bitrate savings between 1.07% and 1.94%.

This paper introduces a method for digitally signing and verifying elementary video bitstreams. The method verifies temporal consistency of the video while allowing random access into the bitstream and adaptation to temporal and spatial scalability including sub-bitstream extraction. It was adopted by the Joint Video Experts Team (JVET) into the Versatile supplemental enhancement information messages for coded video bitstreams (VSEI) specification version 4 by introducing three new Digitally Signed Content SEI messages.

The continuous improvements on image compression with variational autoencoders have lead to learned codecs competitive with conventional approaches in terms of rate-distortion efficiency. Nonetheless, taking the quantization into account during the training process remains a problem, since it produces zero derivatives almost everywhere and needs to be replaced with a differentiable approximation which allows end-to-end optimization. Though there are different methods for approximating the quantization, none of them model the quantization noise correctly and thus, result in suboptimal networks. Hence, we propose an additional finetuning training step: After conventional end-to-end training, parts of the network are retrained on quantized latents obtained at the inference stage. For entropy-constraint quantizers like Trellis-Coded Quantization, the impact of the quantizer is particularly difficult to approximate by rounding or adding noise as the quantized latents are interdependently chosen through a trellis search based on both the entropy model and a distortion measure. We show that retraining on correctly quantized data consistently yields additional coding gain for both uniform scalar and especially for entropy-constraint quantization, without increasing inference complexity. For the Kodak test set, we obtain average savings between 1% and 2%, and for the TecNick test set up to 2.2% in terms of Bjontegaard-Delta bitrate.

In the last years, deep video coding has attracted a lot of research interest. Usually, it employs the concept of inter coding by transmitting features in a latent space that represent a motion field or a residual. However, in such a setting there are still redundancies between the features of consecutive frames. In previous approaches, these redundancies are exploited for compression by adding an additional input at the encoder and decoder. However, this often comes at the cost of changing the whole network architecture. In this paper, we present a conditional coding for motion features which utilizes already transmitted features for coding the features of the current picture in a more effective way. This concept can be applied on top of any existing coding framework. Our coding experiments, which were conducted for JVET test sequences, demonstrate that the proposed conditional motion feature coding can yield bit-rate savings of up to 9% relative to an independent coding of the motion features for individual pictures.

This paper presents a convolutional neural network (CNN)-based enhancement to inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples. It is motivated by the theoretical consideration that neural network-based methods have a higher degree of signal adaptivity than conventional signal processing methods and that spatially neighboring reference samples have the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. We show that adding a polyphase decomposition stage to the CNN results in a significantly better trade-off between computational complexity and coding performance. Incorporating spatial reference samples in the inter prediction process is challenging: The fact that the input of the CNN for one block may depend on the output of the CNN for preceding blocks prohibits parallel processing. We solve this by introducing a novel signal plane that contains specifically constrained reference samples, enabling parallel decoding while maintaining a high compression efficiency. Overall, experimental results show average bit rate savings of 4.07% and 3.47% for the random access (RA) and low-delay B (LB) configurations of the JVET common test conditions, respectively.

Modern hybrid video codecs like Versatile Video Coding (VVC) heavily rely on transform coding tools. Given a prediction signal at the encoder, the residual is transformed using trigonometric transforms. Rate-distortion-optimized quantization (RDOQ) and entropy coding of the transformed residual is well-understood due to the orthogonality and the energy compaction of these transforms. Within this setting, there is considerable success in optimizing secondary orthogonal transforms. The most prominent example is the Low-Frequency Non-Separable Transform (LFNST) in VVC. However, training nonlinear transforms without re-designing the RDOQ and entropy coding stage is a hard problem. In learned image compression, variational autoencoders have shown impressive results, but they use their own entropy model, remain difficult to train for small blocks and RDOQ is nontrivial for them. This paper describes a novel design of a nonlinear transform network for block-based video coding. Given a transform block, a fully-connected neural network predict coefficients from previously reconstructed ones and the adherent block boundary, such that only the residual coefficients need to be transmitted. Furthermore, another neural network filters the entire transform block before the inverse transform is applied and the intra prediction signal is added. Against the Versatile Video Coding Test Model 14.2 (VTM-14.2), luma bit-rate savings of approximately 1.9 % are reported for the All-Intra configuration.

In this paper, we present a novel in-loop filter for video coding which is based on a convolutional neural network (CNN). For that, the adaptive loop filter (ALF) of Versatile Video Coding (VVC) is generalized to define the model architecture for a CNN-based in-loop filter which requires significantly lower computational complexity compared to other existing CNN-based in-loop filters. Experimental results show that, under the JVET common test conditions, BD-rate savings of 1.84% and 1.83% can be achieved compared to VVC for the all-intra and random-access configurations, respectively. At the same time, the computational complexity is at only about 455 multiplications per luma sample.

Data-driven optimization is employed to study alternative approaches [1] to the probability estimator of the the Enhanced Compression Model (ECM) (which includes additional coding tools on top of the Versatile Video Coding standard). In ECM, each context model uses a weighted sum of two hypotheses for probability estimation with different associated adaptation rates. Four alternative approaches are studied:

In August 2022, ISO/IEC MPEG published the first international standard on compression of neural networks, namely Neural Network Coding (NNC, MPEG-7 part 17). It compresses neural networks to about 5% to 15% in size at virtually no performance loss. In NNC, the model weights are usually quantized and then encoded into the bitstream using DeepCABAC entropy coding. In order to improve the coding efficiency, this paper presents new training strategies for optimized model weights considering the quantization and entropy coding process of NNC, by making the training process bitrate- and quantization-aware. With this bitrate-performance optimized training the bitrate can be further reduced by more than 25% on average for state-of-the-art image classification models.

The research on deep-learned end-to-end video compression has attracted a lot of attention over the course of recent years. A central component of many approaches is to perform motion-compensated prediction by using convolutional neural networks (CNN) which determine a compressed representation of the motion field as features. Often, this task is divided into searching motion vectors by one network and efficiently representing them by another one. However, these networks may find motion fields far from optimal because the search radius of CNNs is mainly determined by their depth and kernel size. In this paper, we apply motion estimation techniques from classical block-based hybrid video compression to search a motion field which is then fed into a variational autoencoder. These strategies include different distortion measures, different block partitions and an improved approximation of the residual bitrate. With our modifications, bitrate savings of up to 13% over the underlying end-to-end based video codec can be obtained.

The new Versatile Video Coding standard (VVC) exhibits a significantly higher coding efficiency in comparison to its predecessor High Efficiency Video Coding (HEVC). Since the finalization of VVC in July 2020 many new in-loop compression tools were suggested, especially with data-driven approaches. The question arises if non -data-driven in-loop filtering approaches can further increase coding efficiency beyond the capability of VVC. In this paper we approach the task of filtering images corrupted by quantization noise from the perspective of applied harmonic analysis, an area of applied mathematics focusing on the efficient representation, analysis and encoding of data. The shearlet-based intensity classification loop filter (SCLF) is a novel non-data-driven approach using an overcomplete and sparsifying transform, the shearlet transform. The basic idea is to achieve a signal-noise separation by applying a shearlet transform. Shearlets can identify important structures of natural images such as edges in the sparse transform domain. Each shearlet transform coefficient is classified into different classes. All coefficients with similar intensities are grouped into the same class, which eventually gives a partition of the set of all shearlet coefficients. This separates important information from noise and ensures an accurate reconstruction of original information by performing an inverse shearlet transform combined with Wiener filtering separately for each class. SCLF effectively removes compression artifacts and therefore restores subjective visual quality. Simulation results show that by adding SCLF to the in-loop stage yet further average bit rate reductions of up to 1.5% are achieved for JVET common test sequences.

The performance of variational auto-encoders (VAE) for image compression has steadily grown in recent years, thus becoming competitive with advanced visual data compression technologies. These neural networks transform the source image into a latent space with a channel-wise representation. In most works, the latents are scalar quantized before being entropy coded. On the other hand, vector quantizers generally achieve denser packings of high-dimensional data regardless of the source distribution. Hence, low-complexity variants of these quantizers are implemented in the compression standards JPEG 2000 and Versatile Video Coding. In this paper we demonstrate coding gains by using trellis-coded quantization (TCQ) over scalar quantization. For the optimization of the networks with regard to TCQ, we employ a specific noisy representation of the features during the training stage. For variable-rate VAEs, we obtained 7.7% average BD-rate savings on the Kodak images by using TCQ over scalar quantization. When different networks per target bitrate are optimized, we report a relative coding gain of 2.4% due to TCQ.

Recently, convolutional neural network (CNN)-based in-loop filters have been introduced for video coding and they show huge coding gains. However, one of the main issues of this approach is the high computational complexity of these filters. In this paper, we present various settings for CNN-based in-loop filters targeting on the reduction of their decoder complexity and describe the corresponding gain-complexity trade-offs. For this, we introduce an effective complexity measure and show that it is possible to notably reduce this value for some CNN-based in-loop filters while keeping the compression gain over Versatile Video Coding (VVC).

In this paper, a data-driven generalization of the adaptive loop filter (ALF) of Versatile Video Coding (VVC) is presented. It is shown how the conventional ALF process of classification and FIR filtering can be generalized to define a natural model architecture for convolutional neural network (CNN) based in-loop filters. Experimental results show that over VVC, average bit-rate savings of 3.85%/4.75% and 4.39%/4.33% can be achieved for the all intra and random access configurations in the low- and high-QP settings.

This paper presents an improved probability estimation scheme for the entropy coder of Incremental Neural Network Coding (INNC), which is currently under standardization in ISO/IEC MPEG. More specifically, the paper first analyzes the compression performance of INNC and how the bitstream size relates to the neural network (NN) layers. For the layers requiring the most bits, it analyzes the coded NN weight updates and their temporal dependencies. Major finding is that the probability of a significant (i.e., non-zero) update for a weight can depend considerably on whether the weight has been updated before. Based on this finding, the paper proposes a new probability estimation scheme: Depending on whether a significant update has been received before (i.e., based on the weight’s history), the entropy coder models the probability for a current significant update differently. This scheme achieves a bitstream size reduction of about 2% and 1% in a transfer and a federated learning scenario, respectively, without any accuracy loss or significant complexity increase. Therefore, MPEG adopted our history dependent significance probability (HDSP) scheme to its emerging standard for INNC.

Variational autoencoders have shown promising results for still image compression and have gained a lot of consideration in this field. Recently, noteworthy attempts were made to extend such end-to-end methods to the setting of video compression. Here, low-latency scenarios have been commonly investigated. In this paper, it is shown that the compression efficiency in this setting is improved by applying tools that are typically used in block-based hybrid coding such as rate-distortion optimized encoding of the features and advanced interpolation filters for computing samples at fractional positions. Additionally, a separate motion estimation network is trained to further increase the compression efficiency. Experimental results show that the rate-distortion performance benefits from including the aforementioned tools.

This paper presents a CNN to reduce the encoding time of a VVC-based intra-picture encoder. For encoding a 32 x 32 block, the CNN estimates two partitioning parameters that restrict the allowed coding block width and height. To estimate them such that the encoder skips testing inefficient partitioning modes, we train the CNN as follows: First, we generate training data by encoding sequences without the CNN. While encoding, we test all combinations of the two parameters for each 32 x 32 block and store the resulting Lagrangian rate-distortion-time (RDT) cost. We use the recorded cost to derive the loss function when training the CNN. Consequently, the CNN is trained such that it minimizes the Lagrangian RDT cost. Our CNN reduces the encoding time by 50% with a bit rate increase of 0.9%, which outperforms existing CNN-based approaches. Our generic training approach could also be applied for other encoder parameters.

This paper presents the intra prediction and mode coding of the Versatile Video Coding (VVC) standard. This standard was collaboratively developed by the Joint Video Experts Team (JVET). It follows the traditional architecture of a hybrid block-based codec that was also the basis of previous standards. Almost all intra prediction features of VVC either contain substantial modifications in comparison with its predecessor H.265/HEVC or were newly added. The key aspects of these tools are the following: 65 angular intra prediction modes with block shape-adaptive directions and 4-tap interpolation filters are supported as well as the DC and Planar mode, Position Dependent Prediction Combination is applied for most of these modes, Multiple Reference Line Prediction can be used, an intra block can be further subdivided by the Intra Subpartition mode, Matrix-based Intra Prediction is supported, and the chroma prediction signal can be generated by the Cross Component Linear Model method. Finally, the intra prediction mode in VVC is coded separately for luma and chroma. Here, a Most Probable Mode list containing six modes is applied for luma. The individual compression performance of tools is reported in this paper. For the full VVC intra codec, a bitrate saving of 25% on average is reported over H.265/HEVC using an objective metric. Significant subjective benefits are illustrated with specific examples.

The final version of the Versatile Video Coding (VVC) standard incorporates the Matrix-Based Intra Prediction (MIP) tool. It consists of additional intra prediction modes which were derived from a data-driven training. These modes, in general, are applied to the luma component only. This paper describes how to apply the MIP modes to the chroma components in certain cases. In the generic case, if a chroma block uses the intra direct mode (DM), its intra mode is derived from the co-located luma block. If this luma block uses MIP, the chroma block uses the planar mode, because a MIP mode is not applicable for all block shapes. However, if the chroma format is 4:4:4 and single-tree coding is used, luma and chroma blocks share the same partitioning. The paper investigates the impact of using the MIP mode of the luma component when the chroma mode is the DM in these cases. This extension also harmonizes MIP with the Adaptive Color-Space Transform (ACT) for RGB-content, because the chroma mode of blocks using the ACT is inferred to be the DM. For the extended MIP tool in the Versatile Video Coding Test Model 11.2 (VTM-11.2) that implements the final VVC standard, bit-rate savings of 1.51%/0.60%/0.62% in terms of the Bj⊘ntegaard-Delta bit rate (BD-rate) are reported compared to the VTM-11.2 with MIP disabled for the All-Intra configuration and natural content in the RGB 4:4:4 format.