2024 Cycle 2 IEEE SPS Chapter Initiative on Multimedia Signal Processing

Trends on Image and Video Coding Standardization

Fernando Pereira (Instituto Superior Técnico / Instituto de Telecomunicações)

Abstract: Visual communications have a fundamental role in Human societies. In the digital era, this has led to the explosion of image and video-based applications and services, notably following the democratization of image and video acquisition, storage and streaming. However, conventional image and video-based experiences are far from the real-world immersion; this has motivated a rush for more realistic, immersive and interactive visual experiences, notably offering 6-DoF (Degrees-Of-Freedom) experiences where the user is free to exploit the six translational and rotational types of motion, possible in the real world.

Recent years have seen a growing wave of data driven algorithms, such as deep neural networks, which have taken a lead role in many research and development areas, especially in multimedia. This interest is driven by several factors, notably advances in processing power, availability of large data sets, and algorithmic advances. Deep learning (DL)-based solutions are the state-of-the-art for multiple computer vision tasks, both high- and low-level. These advances have led to the exploitation of DL-based tools in the visual coding domain, reaching competitive performance in a short time.

The potential of this type of coding approach has been recognized by JPEG and MPEG which have already developed DL-based coding standards, and more are coming. These standards demonstrate that, after many years of developments and advances with hand-crafted coding tools, the visual coding landscape is facing a revolution with the emergence of DL-based technology, which has the potential to unify several visual modalities under the same technological umbrella, offering compressed visual representations able to simultaneously and effectively serving humans and machines. In this context, this talk will address the recent trends on image and video coding standardization.

Bio: FERNANDO PEREIRA (Fellow, IEEE) is currently a Professor with the Department of Electrical and Computers Engineering, Instituto Superior Técnico, Universidade de Lisboa, and a Senior Researcher with the Instituto de Telecomunicações, Lisbon, Portugal. He was an IEEE Distinguished Lecturer, in 2005, and elected as an IEEE Fellow, in 2008, for contributions to object-based digital video representation technologies and standards. Since 2013, he has also been a EURASIP Fellow for contributions to digital video representation technologies and standards. He is/has been a member of the Scientific and Program Committees of many international conferences and workshops. He has been elected to serve on the Signal Processing Society Board of Governors in the capacity of Member-at-Large for the 2012 and 2014-2016 term. He has served as the SPS Vice-President for Conferences during the 2018-2020 term. In 2024 he received the IEEE SPS Leo L. Beranek Meritorious Service Award. From 2013 to 2015, he was the Editor-in-Chief of the IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING. He has contributed more than 300 articles in international journals, conferences, and workshops, and made several tens of invited talks at conferences and workshops.

He has participated in the MPEG standardization activities, notably as Requirements Sub-group Chair, and the Chair of many Ad Hoc Groups related to the MPEG-4 and MPEG- 7 standards. Since February 2016, he has been the JPEG Requirements Sub-group Chair. His research interests include visual data analysis, coding, description, adaptation, quality assessment, and advanced multimedia services.

Visual Quality Assessment: Current Practices and Challenges

António Pinheiro (University of Beira Interior / Instituto de Telecomunicações)

Abstract: The recent developments on high-quality equipment for images and video visualization created a general demand for novel models of quality assessment. Typically, image and video representation methods use lossy compression models that produce visible distortions, which affect the perceived experience. For the quality evaluation of such distortions, subjective quality evaluations are usually conducted, followed by the adoption of objective metrics validated upon the subjective evaluations.

A generic description of the current quality assessment methodologies will be given, and some limitations will be identified. Those limitations are the basis of current research on visual quality assessment, leading to new subjective quality evaluation approaches that target near-visually lossless distortion evaluation or models for quality evaluation of immersive and 3D visual data. Current challenges will be identified.

Bio: Antonio Pinheiro is an Associate Professor at UBI (Universidade da Beira Interior), and a researcher at IT (Instituto de Telecomunicações), Portugal. He received the "Licenciatura" from IST, Lisbon in 1988 and the PhD in Electronic Systems Eng. from University of Essex in 2002. Currently, his research interests are on Multimedia Quality Evaluation and Emerging 3D Imaging Technologies. He is a Portuguese delegate to ISO/IEC JTC1/SC29 and the Communication Subgroup chair of JPEG. He was host of the 100th JPEG meeting. He is Associate editor of IEEE Trans. on Multimedia, member of the IEEE SPS MMSP TC, and a senior member of IEEE.

Learning-based Point Cloud Coding

Nuno Rodrigues (Polytechnic of Leiria / Instituto de Telecomunicações)

Abstract: The demand for more immersive and interactive experiences has driven the use of new 3D multimedia formats, more fitted to virtual and augmented reality applications. Among these formats, point clouds (PCs) have gained relevance, due to their ability to represent the scenes’ 3D visual information, using a set of points and associated attributes, notably color. To offer realistic and immersive experiences, PCs need millions, or even billions, of points, thus asking for efficient representation and coding solutions, which are critical for the practical deployment of emerging applications and services.

This presentation will describe the state-of-the-art of point cloud coding, with a focus on the potential of learning-based coding solutions, including a description of the new JPEG Pleno Point Cloud Coding (PCC) standard. The presentation compares the compression approaches and performance of learning-based PCC methods with previous conventional PCC solutions. Furthermore, the presentation highlights the potential of learning-based coding methods to offer an effective, common, unique representation for both human visualization and machine consumption. By using compressed domain processing techniques, which use the same features extracted by the PC encoder which are used for decoding the PC for human visualization, computer vision algorithms avoid the traditional performance and complexity penalties associated with the decoding process.

Bio: Nuno Rodrigues is a Professor with the Department of Electrical Engineering, School of Technology and Management, Polytechnic University of Leiria, Portugal, and a Senior Researcher with Instituto de Telecomunicações, Portugal. He is a Senior member of IEEE and the Chair of the Signal Processing Chapter of IEEE Portugal. He serves or has served as an Associate Editor for IEEE Transactions on Multimedia, IEEE Open Journal on Signal Processing, EURASIP Journal on Image and Video Processing and Signal Processing: Image Communication.

He has coordinated and participated as a Researcher in various national and international funded projects. He has supervised three concluded Ph.D. theses and several M.Sc. theses. He is the coauthor of one book and more than 100 publications, including book chapters and papers in national and international journals and conferences. His research interests include topics related with image and video coding and processing, for different signal modalities and applications. He has been part of the JPEG Pleno project and contributed to the development of the JPEG Pleno Point Cloud Coding standard, as well as to several other deep learning-based techniques for point cloud coding and processing.

Emerging 3D Representation Models: NeRFs and Splatting

André Guarda (Instituto de Telecomunicações)

Abstract: The emergence of sophisticated 3D representation models has revolutionized the field of computer vision and graphics, providing unprecedented capabilities in rendering and understanding complex scenes. This presentation delves into two cutting-edge methodologies: Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting, both of which have shown remarkable potential in creating highly detailed and realistic 3D reconstructions.

NeRFs leverage neural networks to represent a scene as a continuous 3D volume, enabling high-fidelity rendering from sparse and unstructured image collections. By optimizing the parameters of the neural network, NeRFs can synthesize novel views of a scene with stunning accuracy and detail. This technique has significantly impacted various applications, from virtual reality to photorealistic rendering in visual effects.

On the other hand, 3D Gaussian Splatting offers a novel approach to 3D representation by utilizing a set of Gaussian functions to model the geometry and appearance of objects. This method provides a flexible and efficient framework for capturing the nuances of a scene, including fine details and complex lighting effects. 3D Gaussian Splatting has shown promise in scenarios where traditional mesh-based approaches fall short, particularly in representing soft and translucent objects.

This presentation will explore the underlying principles of NeRFs and 3D Gaussian Splatting, highlighting their unique advantages and limitations, demonstrating how these models are transforming 3D representations, and discussing future directions and potential applications.

Bio: André Guarda received his B.Sc. and M.Sc. degrees in electrotechnical engineering from Instituto Politécnico de Leiria, Portugal, in 2013 and 2016, respectively, and the Ph.D. degree in electrical and computer engineering from Instituto Superior Técnico, Universidade de Lisboa, Portugal, in 2021.

He has been a researcher at Instituto de Telecomunicações since 2011, where he currently holds a Post-Doctoral position. His main research interests include multimedia signal processing and coding, with particular focus on point cloud coding with deep learning. He has authored several publications in top conferences and journals in this field, and is actively contributing to the standardization efforts of JPEG and MPEG on learning-based point cloud coding.

He was a founding member of the IEEE Student Branch of Instituto Politécnico de Leiria, and currently serves as the Treasurer of the IEEE Signal Processing Chapter in Portugal.

Light Field Coding: Past, Present and Future

Eduardo da Silva (Federal University of Rio de Janeiro)

Abstract: Light fields are a simplification of the 7D plenoptic function in which the visual information can be encoded as a matrix of views. As such, they inherently generate a large amount of data, which asks for efficient coding techniques. This talk will briefly discuss the past, present, and future of light field coding. We start with pseudo-video-based techniques using efficient video coders and multiview extensions of video coding algorithms. Then, we will pass through the interview prediction methods based on view synthesis techniques. Techniques that directly exploit the 4D redundancy of light fields, such as the ones based on 4D transforms, will then be dealt with. The talk ends with an overview of learning-based methods, including the ones based on implicit representations.

Bio: Eduardo A. B. da Silva was born in Rio de Janeiro, Brazil. He received the Electronics Engineering degree from Instituto Militar de Engenharia (IME), Brazil, in 1984, the M.Sc. degree in Electrical Engineering from Universidade Federal do Rio de Janeiro (COPPE/UFRJ) in 1990, and the Ph.D. degree in Electronics from the University of Essex, England, in 1995. He has been a Universidade Federal do Rio de Janeiro professor since 1989. He has co-authored four books, including ”Digital Signal Processing - System Analysis and Design,” published by Cambridge University Press, with English, Chinese, and Portuguese editions. His research interests lie in signal and image processing, signal compression, 3D videos, computer vision, light fields, machine learning, and its applications to telecommunications and the oil and gas industry. He is currently the Requirements Vice Chair of the Joint Photographic Experts Group from ISO/IEC and was editor of the standard ISO/IEC CD 21794-2, JPEG Pleno Plenoptic image coding system. In 2023, he was awarded the ISO Excellence Award for his leadership in the Light Field Compression standardization process. He has been a member of the Brazilian Academy of Engineering (ANE) since 2023 and a member of the Brazilian Academy of Sciences since 2024.

Light Field Processing for Immersive Systems

Paulo Nunes (University Institute of Lisbon / Instituto de Telecomunicações)

Abstract: Light Field (LF) imaging is a promising solution for providing more immersive and closer to reality multimedia experiences to end-users with unprecedented creative freedom and flexibility for applications in different areas, such as Virtual Reality (VR) and Augmented Reality (AR), cinematography, three-dimensional (3D) television, biometric recognition, and medical imaging.

LFs enable capturing both light intensity and direction information of a scene in a Four-Dimensional (4D) array, known as the 4D LF. The rich information included in 4D LFs enables the viewer to explore the scene from different perspectives, hence, enhancing depth perception and realism. However, the promise of an immersive experience comes with challenges that need to be investigated, notably in terms of processing and editing. One example of those challenges is to efficiently process/edit 4D LFs by exploiting the spatio-angular correlations while ensuring spatial accuracy and angular consistency. This challenge is tackled through several methods that together form a pipeline to efficiently process/edit 4D LFs.

At first, an efficient disparity propagation method that enables computing angularly consistent disparity maps for all LF views is presented. Afterwards, novel over-segmentation methods that rely on disparity maps as an additional guiding feature to group corresponding pixels across LF views into spatio-angular segments are briefly described. To illustrate the benefits of these solutions, over-segmented 4D LFs are then used as an intermediate representation that enables LF segmentation and facilitates neural style transfer propagation.

Finally, it will be shown that representing 4D LFs based on over-segmentation allows the usage of classical graph cut and graph neural networks to achieve efficient and angularly consistent 4D LF mid-level segmentation.

In summary, this talk presents recent advances and possible future research directions on the processing of this immersive imaging modality that is beyond the traditional 2D images and provides a more realistic experience to the end users.

Bio: Paulo Nunes was born in Lisbon, Portugal, in August 1969. He graduated in Electrical and Computer Engineering from Instituto Superior Técnico and received the MSc and PhD degrees in 1992, 1995 and 2007, respectively.

In 2000, he joined the Department of Information Sciences and Technologies (DCTI) at the University Institute of Lisbon (ISCTE-IUL), where he has been an Associate Professor since 2020, having taught various courses in the areas of computer networks, computer architecture, multimedia communications and signal processing. He has supervised 4 PhD theses, one of which received the IBM Scientific Award in 2016, and 14 MSc theses. In addition to teaching, he has also played an active role in university management, such as Director of the Degree in Telecommunications and Computer Engineering, Sub-director and elected member of the Scientific Committee of the DCTI at ISCTE-IUL. In terms of scientific activity, most of his work has been carried out in the Multimedia Signal Processing Group at Instituto de Telecomunicações, where he has been a researcher since 1995. His main areas of scientific work have been image and video processing and coding, including new visual modalities such as light field imaging. To date, he has published 18 papers in international journals, 8 book chapters, 42 papers in international conferences and 13 papers in national conferences.

He also played a very active role in the work of the MPEG committee during the development and definition phase of the MPEG-4 visual standard. In addition, over the last few years, he has actively participated in a total of 13 funded projects (6 European and 7 national), having been coordinator of the Portuguese participation in one EU project and PI in two national projects.

He has regularly served as a reviewer for various international journals and conferences, and participated in the organization of several international conferences and special sessions, having been the Local Organizing Committee Chair and Co-Technical Program Chair of 3DTV-Con conference held in Lisbon in July 2015. He was also the secretary of CT 120 from 2009 to 2017.

He has been a member of the ISCTE Choir, as Tenor, since September 2016.

Biometrics: making sure it is you

Paulo Correia (Instituto Superior Técnico / Instituto de Telecomunicações)

Abstract: This presentation provides an introduction to biometric recognition, discussing the main biometric modalities used, and discussing the vulnerabilities of biometric recognition systems. It concludes with an overview of other applications of biometrics, and a discussion of open issues.

Bio: Paulo Lobato Correia is Professor at the Department of Electrical and Computer Engineering, Instituto Superior Técnico, Universidade de Lisboa, Portugal. He is Senior Researcher of the Multimedia Signal Processing research group of Instituto de Telecomunicações, and Senior Member of the IEEE.

Paulo Correia coordinated participation in several research projects dealing with image and video analysis and processing. For instance, he was Vice-Chair of the Management Committee of the European Cooperation in Science and Technology (COST) Action IC1106: "Integrating Biometrics and Forensics for the Digital Age" (2014–2016).

He was Editor in Chief of IET Biometrics for the term 2020–2023 and Area Editor (for Multimedia) of the Elsevier Signal Processing Journal (2018–2020). He was Associate Editor of the IEEE Transactions on Circuits and Systems for Video Technology (2006–2014), of the Elsevier Signal Processing Journal (2005–2017), and of IET Biometrics (2013–2019). He has been Guest Editor of several special issues for scientific journals such as IET Biometrics or Elsevier Pattern Recognition Letters, and cooperated in many conference organizing committees, as general chair, program chair, finance chair or special sessions chair, among other roles.

He is a founding member of the Advisory Board of the European Signal Processing Association (EURASIP), and was the elected chairman of EURASIP’s Technical Area Committee on “Biometrics, Data Forensics and Security” for the term 2018–2020.

He is president of the Portuguese standardization committee on Biometrics (CT226), for the period 2023-2024.

He has co-authored more than 150 journal and conference papers. The main research interests are about video analysis and processing, with emphasis on biometric signal analysis, targeting recognition, sports, medical and forensic applications.

For more detail please visit: https://www.it.pt/Members/Index/390

Medical Imaging: Coding and Processing

Sérgio Faria (Polytechnic of Leiria / Instituto de Telecomunicações)

Abstract: The rapid advancement of medical imaging technologies is pushing the limits of our ability to process, store and transmit the vast amount of data they generate. This presentation highlights some contributions to the development of medical image compression techniques for 2D, volumetric and multimodal datasets, as well as innovations in automatic data processing using machine learning.

Bio: Sérgio Faria (Senior Member, IEEE) was born in Horta, Portugal, in 1965. He received the Ph.D. degree in electronics and telecommunications from the University of Essex, U.K., in 1996. He has been a Full Professor with the Department of Electrical Engineering, Polytechnic of Leiria, since 2016. He was the Former Vice-President of the Polytechnic of Leiria, from 1997 to 1999. He was the Head of the IT Leiria Site, in 2014, and the Former Head of the Multimedia Signal Processing Group, Leiria. His research interests include 2D/3D image and video processing and coding and medical imaging. He is the co-inventor of two patents, co-founded a Tech startup, coauthor of one book, 14 book chapters, and more than 200 publications in international journals and conferences. He participates in the MPEG and JPEG activities and is currently an Associate Editor of IEEE TRANSACTIONS ON IMAGE PROCESSING and an Associate Editor of Signal Processing: Image Communication (Elsevier). He received the title of Agregado from IST, University of Lisbon, in 2014.