Post-doctoral researcher; Hybrid Implicit Neural Representations for Next-Generation Learned Video Compression

Join InterDigital's Video Lab to pioneer the next generation of energy-efficient AI for video. As a postdoctoral researcher, you will help shape technologies that reduce the environmental footprint of generative AI while contributing to international standards and publishing cutting-edge research.

About InterDigital

InterDigital is a global research and development company focused primarily on wireless, video, artificial intelligence (“AI”), and related technologies. We design and develop foundational technologies that enable connected, immersive experiences in a broad range of communications and entertainment products and services. We license our innovations worldwide to companies providing such products and services, including makers of wireless communications devices, consumer electronics, IoT devices, cars and other motor vehicles, and providers of cloud-based services such as video streaming. As a leader in wireless technology, our engineers have designed and developed a wide range of innovations that are used in wireless products and networks, from the earliest digital cellular systems to 5G and today’s most advanced Wi-Fi technologies. We are also a leader in video processing and video encoding/decoding technology, with a significant AI research effort that intersects with both wireless and video technologies. Founded in 1972, InterDigital is listed on Nasdaq.

InterDigital is a registered trademark of InterDigital, Inc.

For more information, visit: www.interdigital.com .

Background and Motivation

The continuous growth of video traffic generated by streaming platforms, immersive media, cloud gaming, autonomous systems, and machine vision is placing unprecedented pressure on communication and storage infrastructures. Although conventional video codecs such as H.266/VVC, AV1 and HEVC continue to improve coding efficiency through decades of algorithmic refinements, their block-based architecture is approaching diminishing returns. Their design relies on handcrafted signal processing modules — including prediction, transforms, quantization, in-loop filtering and entropy coding — that have been carefully optimized but remain fundamentally constrained by assumptions about natural image statistics.

Learned Video Compression (LVC) has emerged during the past decade as a promising alternative. Instead of optimizing individual codec modules independently, learned codecs formulate compression as a global end-to-end optimization problem. Neural networks jointly learn the analysis transform, synthesis transform, entropy model and motion representation by minimizing a rate-distortion objective where (D) measures reconstruction distortion while (R) estimates the coding rate through differentiable entropy models.

Recent learned image and video codecs have demonstrated compression performances comparable to — or surpassing — the latest standard codecs for several operating points. More importantly, they provide a flexible framework in which every component can be jointly optimized for a target application.

Despite these impressive advances, several important challenges remain before learned video codecs can become a practical replacement for conventional standards.

First, current architectures remain computationally demanding. Deep convolutional encoders, decoders and optical-flow estimation networks require millions of parameters and substantial computational resources during both training and inference.

Second, entropy modeling remains one of the principal bottlenecks. State-of-the-art autoregressive models improve coding efficiency but introduce strong sequential dependencies that significantly reduce decoding parallelism.

Third, current codecs still represent images and videos through discrete latent feature maps. While highly effective, these representations inherit many limitations of grid-based signals, including resolution dependence, interpolation artifacts and limited geometric adaptability.

Finally, temporal redundancy is often modeled explicitly through optical flow or recurrent networks, both of which become increasingly expensive for long sequences or high-resolution content.

These limitations motivate the exploration of alternative signal representations capable of providing richer spatial and temporal modeling while maintaining efficient entropy coding.

Hybrid Implicit Neural Representations

Implicit Neural Representations (INRs) have recently emerged as a fundamentally different paradigm for representing visual signals. Rather than storing an image or a video as discrete samples on a grid, an INR represents the signal as a continuous function where the neural network maps spatial (or spatio-temporal) coordinates

X Χ directly to signal values Υ Υ.

Continuous representations naturally provide:

arbitrary-resolution reconstruction,

smooth interpolation,

compact parameterization,

geometric continuity.

However, pure coordinate-based INRs exhibit several limitations for compression. They often require relatively large neural networks, suffer from slow convergence, and struggle to reproduce highly textured regions without substantially increasing model complexity.

Hybrid Implicit Neural Representations (Hybrid INRs) have recently addressed these limitations by combining learned latent grids with coordinate-based neural decoders.

Instead of relying solely on coordinates, the decoder receives both

local latent features extracted from hierarchical latent grids,

continuous spatial coordinates.

The latent grids capture high-frequency local structures while the implicit decoder reconstructs the continuous signal. Hierarchical latent resolutions allow coarse image structures to be represented at low spatial resolutions and fine details at higher resolutions.

Among recent examples, Cool-Chic [Ladune et al., 2003] and C3 [Kim et al., 2024] ha ve demonstrated that hybrid INRs constitute an extremely competitive framework for learned image compression. Its hierarchical latent representation, lightweight implicit decoder and learned entropy model achieve state-of-the-art rate-distortion performance while requiring significantly fewer decoder parameters than conventional CNN-based architectures.

The advantages of hybrid INRs include:

continuous image representation,

compact decoder architectures,

scalable latent hierarchies,

compatibility with autoregressive entropy models,

excellent adaptation to image geometry.

Nevertheless, current hybrid INR methods remain largely restricted to still images. Extending this framework to video introduces new scientific questions regarding temporal representation, temporal prediction and entropy modeling [Leguay et al., 2024] .

Extending Hybrid INRs to Learned Video Compression

The central objective of this project is to develop a new generation of learned video codecs based on spatio-temporal hybrid implicit neural representations. Instead of treating each frame independently, the proposed framework will model an entire video sequence as a continuous function where the decoder jointly exploits spatial coordinates, temporal coordinates and hierarchical latent representations. T he key challenge lies in introducing temporal redundancy into the latent hierarchy without sacrificing the compactness that makes hybrid INRs attractive. Several complementary research directions will be investigated.

Hierarchical Spatio-Temporal Latent Representation

Current hybrid INR codecs organize latent grids across multiple spatial resolutions. This project proposes extending t he representation to the temporal dimension. This could include temporal latent pyramids, key-frame and inter-frame latent decomposition, multi-scale temporal feature grids, adaptive temporal resolutions depending on motion complexity …

Continuous Temporal Prediction

Conventional learned video codecs typically estimate optical flow between successive frames. Hybrid INRs provide an opportunity to replace ex