Submitted to IEEE Transactions on Information Forensics and Security (T-IFS). Under revision.

Chroma from Luma (CfL) prediction is an efficient coding tool in AV1 which builds chroma prediction by implementing a linear model on luma pixels. To avoid transmission, the off-set factor in the linear model is set to the average of neighboring chroma pixels. An improved CfL algorithm is proposed to derive the offset factor based on a virtual chroma block. Such a block is constructed by using the chroma of the matched pixel which is determined according to the luma difference of neighboring regions and the co-located luma. Compared with libaom, the proposed CfL algorithm provides 0.31% and 0.15% weighted PSNR BD-rate saving under AI and RA configuration, respectively. Experimental results show that over 1.00% and 0.80% BD-rate saving can be achieved for chroma components under AI and RA configuration. With the proposed algorithm, the percentage of pixels with CfL as the optimal coding mode is increased.

The AV1 video compression format is developed by the Alliance for Open Media consortium. It achieves more than a 30% reduction in bit rate compared to its predecessor VP9 for the same decoded video quality. This article provides a technical overview of the AV1 codec design that enables the compression performance gains with considerations for hardware feasibility.

In 2018, the Alliance for Open Media (AOMedia) finalized its first video compression format AV1, which is jointly developed by the industry consortium of leading video technology companies. The main goal of AV1 is to provide an open source and royalty-free video coding format that substantially outperforms state-of-the-art codecs available on the market in compression efficiency while remaining practical decoding complexity as well as being optimized for hardware feasibility and scalability on modern devices. To give detailed insights into how the targeted performance and feasibility is realized, this paper provides a technical overview of key coding techniques in AV1. Besides, the coding performance gains are validated by video compression tests performed with the libaom AV1 encoder against the libvpx VP9 encoder. Preliminary comparison with two leading HEVC encoders, x265 and HM, and the reference software of VVC is also conducted on AOM's common test set and an open 4k set.

Entropy coding is a lossless data compression technique that is widely applied in video codecs to encode syntax elements into bitstreams. Efficient entropy coding requires accurate prediction of the probability distribution of the encoded symbols. In AV1, multi-symbol arithmetic coding is adopted. The symbol probability is derived with handcrafted context models and lookup tables that store the predicted probabilities corresponding to different entropy contexts. The lookup table based scheme has some fundamental deficiencies. The entropy context features have to be discrete so that they can be used to index the lookup tables. To reduce the size of the lookup table, the number of contexts cannot be very large. Moreover, the probability distributions stored in the lookup tables are maintained separately without taking their correlations into consideration. In this paper, we propose a machine learning based scheme that achieves more accurate symbol probability prediction for entropy coding. The proposed approach is implemented in AV1 for the entropy coding of intra prediction modes. Experimental results demonstrate that it can improve the efficiency of entropy coding significantly.

The AV1 codec added a new in-loop super-resolution mode that allows a frame to be encoded at a lower resolution, and then super-resolved normatively to the full resolution, before updating the reference buffers. While encoding at lower resolution and super-resolving to a higher resolution is not a new concept, this is the first time such a mode has been normatively incorporated in a standardized video codec. To this end, AV1 has not only added support for across-scale motion prediction to allow predicting a lower resolution version of a frame from higher resolution reference buffers, but also made various simplifications to the super-resolving process itself after reconstruction to make it both software and hardware-friendly in implementation. Specifically, the super-resolving process in AV1 comprises of normative linear upscaling, followed by a restoration operation to recover the high frequencies using another AV1 tool called loop-restoration that includes a Wiener or Self-guided filter selected in a block switchable manner. Further, in order to enable a cost-effective hardware solution with limited line-buffers, this mode only allows the upscaling/downscaling operation to be horizontal. In this paper, we provide the details of the super-resolution mode in AV1 and some results showcasing the benefits of the same.

Conventional intra prediction modes in image and video coding generate an estimation of a target block by copying or projecting its causal neighboring pixels along certain angles. Such simple directional model does not work well for complex image structures. A set of context-adaptive intra prediction modes based on recursive filtering is proposed in this paper. The prediction of a block is generated by applying linear filtering over certain previously reconstructed or predicted pixels in the causal neighborhood of each pixel recursively. The filter coefficients are estimated with least squares optimization using previously reconstructed pixels in the above and/or left regions of the current block. The configurations for the filters such as filter taps, position of reference pixels, as well as the location and shape of the training regions are all flexible, making the proposed prediction modes highly adaptive to local image texture contexts. A data-driven approach is used to select the optimal subset of all the possible filter configurations while retaining as much coding gains as possible. The proposed approach is tested on the state-of-the-art AV1 video coding standard. AV1 supports sophisticated intra prediction tools such as recursive filtering, quadratic interpolation filtering, intra block-copy, and the palette mode. Experimental results show that the context-adaptive recursive-filtering-based intra prediction modes can achieve significant improvement in compression efficiency.

With more complex partitioning structures in recent generations of video coding standards, the computation complexity of video encoder for partition block size search has been increasing drastically. To expedite the overall encoding process, it is desired to make faster partitioning decisions without much compression performance degradation. In this paper, we propose a multi-scale multi-stage machine learning(ML) based framework to accelerate partition block size search. The framework includes a collection of ML models, each dedicated to make a simple decision for a particular block size at a particular stage during the partitioning rate-distortion optimization(RDO) process. The ML models can predict whether the RD evaluation of certain partition block sizes can be skipped, saving unnecessary computation in the encoder. The proposed approach is implemented and tested on VP9 with the open source library libvpx. Significant encoding speed improvement has been observed with neglectable compression performance regression. The framework and methodology can be easily applied to other video codecs and implementations as well.

AV1 is the state-of-the-art open and royalty-free video compression format that achieves significant bitrate savings over previous generation of video codecs. One of AV1's major improvement over its predecessor VP9 is the support of more diverse and flexible transform size and kernel selection. However, it also drastically increases the search space for transform unit rate-distortion optimization in AV1 encoders. Unlike conventional encoder speed features that are based on heuristics, we propose a machine learning (ML) based approach to accelerate the transform size and kernel search for AV1. The ML models use input features extracted from the prediction residue block such as standard deviation, correlation and energy distribution. The output of the models indicates the estimated likelihood of which transform size and kernel would be selected as the optimal choice. Based on the ML models, the encoder can prune out the transform size and kernel candidates that are unlikely to be selected and save unnecessary computation to compute their rate-distortion cost. The proposed approach is implemented and tested on the AV1 reference library libaom. The experimental results show that satisfactory encoding speed improvement can be achieved with extremely low compression performance loss. The framework and methodology can also be easily migrated to other video codecs and implementations.

Screen content coding plays an important role in many applications. To meet the growing demands of screen content coding, the emerging AV1 video codec incorporates several coding tools, which are specially designed for screen content utilizing its distinctive characteristics. Among these tools, the intra block copy utilizes the characteristic that repeating patterns frequently occur in screen content. This paper presents the technology of intra block copy in AV1. In particular, to efficiently search the predictor in the reconstructed regions of the current picture, AV1 uses the hash matching method at the encoder side. For the generation of hash table, a bottom-to-up manner is adopted to reduce the redundant computation and then decrease the encoding time. In addition, several constraints are involved to facilitate hardware design. Experimental results demonstrate that the intra block copy in AV1 can bring 27.1% bitrate saving for screen content. When compared with the non hash-based intra block copy, the hash-based method achieves 12.2% bitrate saving.

AV1 is an emerging open-source and royalty-free video compression format, which is jointly developed and finalized in early 2018 by the Alliance for Open Media (AOMedia) industry consortium. The main goal of AV1 development is to achieve substantial compression gain over state-of-the-art codecs while maintaining practical decoding complexity and hardware feasibility. This paper provides a brief technical overview of key coding techniques in AV1 along with preliminary compression performance comparison against VP9 and HEVC.

Google started the WebM Project in 2010 to develop open source, royalty- free video codecs designed specifically for media on the Web. The second generation codec released by the WebM project, VP9, is currently served by YouTube, and enjoys billions of views per day. Realizing the need for even greater compression efficiency to cope with the growing demand for video on the web, the WebM team embarked on an ambitious project to develop a next edition codec AV1, in a consortium of major tech companies called the Alliance for Open Media, that achieves at least a generational improvement in coding efficiency over VP9. In this paper, we focus primarily on new tools in AV1 that improve the prediction of pixel blocks before transforms, quantization and entropy coding are invoked. Specifically, we describe tools and coding modes that improve intra, inter and combined inter-intra prediction. Results are presented on standard test sets.

Google started the WebM Project in 2010 to develop open source, royalty-free video codecs designed specifically for media on the Web. The second generation codec released by the WebM project, VP9, is currently served by YouTube, and enjoys billions of views per day. Realizing the need for even greater compression efficiency to cope with the growing demand for video on the web, the WebM team embarked on an ambitious project to develop a next edition codec, VP10, that achieves at least a generational improvement in coding efficiency over VP9. Starting from VP9, a set of new experimental coding tools have already been added to VP10 to achieve decent coding gains. Subsequently, Google joined a consortium of major tech companies called the Alliance for Open Media to jointly develop a new codec AV1. As a result, the VP10 effort is largely expected to merge with AV1. In this paper, we focus primarily on new tools in VP10 that improve coding of the prediction residue using transform coding techniques. Specifically, we describe tools that increase the flexibility of available transforms, allowing the codec to handle a more diverse range or residue structures. Results are presented on a standard test set.

Google started an opensource project, entitled the WebM Project, in 2010 to develop royaltyfree video codecs for the web. The present generation codec developed in the WebM project called VP9 was finalized in mid2013 and is currently being served extensively by YouTube, resulting in billions of views per day. Even though adoption of VP9 outside Google is still in its infancy, the WebM project has already embarked on an ambitious project to develop a next edition codec VP10 that achieves at least a generational bitrate reduction over the current generation codec VP9. Although the project is still in early stages, a set of new experimental coding tools have already been added to baseline VP9 to achieve modest coding gains over a large enough test set. This paper provides a technical overview of these coding tools.

The electric network frequency (ENF) signal can be captured in multimedia recordings due to electromagnetic influences from the power grid at the time of recording. Recent work has exploited the ENF signals for forensic applications, such as authenticating and detecting forgery of ENF-containing multimedia signals, and inferring their time and location of creation. In this paper, we explore a new potential of ENF signals for automatic synchronization of audio and video. The ENF signal as a time-varying random process can be used as a timing fingerprint of multimedia signals. Synchronization of audio and video recordings can be achieved by aligning their embedded ENF signals. We demonstrate the proposed scheme with two applications: multi-view video synchronization and synchronization of historical audio recordings. The experimental results show the ENF based synchronization approach is effective, and has the potential to solve problems that are intractable by other existing methods.

Multiple videos capturing the same scene from possibly different viewing angles may be synthesized for novel immersive experience. Synchronization is an important task for such applications involving multiple pieces of audio-visual data. In this work, we exploit the electric network frequency (ENF) signal inherently embedded in the soundtrack and/or image sequence of video to temporally align video recordings. ENF is the supply frequency of power distribution networks in a power grid. Its value fluctuates slightly from its nominal value of 50 Hz or 60 Hz, and the fluctuation trends stay consistent within the same grid. Audio and video recordings that are created in areas of electric activities may capture the ENF signal due to electromagnetic interferences and other physical phenomena. We propose to synchronize video recordings by aligning the embedded ENF signals. Without major constraints on viewing angle and camera calibration as many existing methods impose, the proposed approach emerges as a new synchronization modality.

The template matching prediction is an established approach to intra-frame coding that makes use of previously coded pixels in the same frame for reference. It compares the previously reconstructed upper and left boundaries in searching from the reference area the best matched block for prediction, and hence eliminates the need of sending additional information to reproduce the same prediction at decoder. In viewing the image signal as an auto-regressive model, this work is premised on the fact that pixels closer to the known block boundary are better predicted than those far apart. It significantly extends the scope of the template matching approach, which is typically followed by a conventional discrete cosine transform (DCT) for the prediction residuals, by employing an asymmetric discrete sine transform (ADST), whose basis functions vanish at the prediction boundary and reach maximum magnitude at far end, to fully exploit statistics of the residual signals. It was experimentally shown that the proposed scheme provides substantial coding performance gains on top of the conventional template matching method over the baseline.