Enhanced residue prediction for Lossless coding of multi- modal image pairs based on image to image translation
Nicolau, D.
;
Parracho, O.
;
Thomaz, L. A.
;
Távora, L.M.
;
Faria, S.M.M.
Enhanced residue prediction for Lossless coding of multi- modal image pairs based on image to image translation, Proc IEEE European Workshop on Visual Information Processing - EUVIP, Gjøvik, Norway, Vol. , pp. - , September, 2023.
Digital Object Identifier: 10.1109/EUVIP58404.2023.10323046
Abstract
Multimodal medical imaging combine data obtained from multiple techniques simultaneously, yielding more detailed information about the content, which is a clear advantage over independent acquisition techniques. As these images are acquired using different imaging modalities and some- times even in different dimensions, they commonly require a geometrical registration process. However, when they are en- coded using standard image codecs the prediction methods do not exploit the redundancies related to the multimodal acqui- sition. In this paper, a novel lossless multimodal prediction module is introduced. The proposed method employs a deep learning-based approach with Image-to-Image translation for the purpose of joint coding of Positron Emission Tomography (PET) and Computed Tomography (CT) image pairs. Prior to the coding stage, a Generative Adversarial Network (GAN) is used for multimodal image translation. Then, a weighted estimated image is utilised as the I-frame, while the weighted sum of the original and synthesised image from the same modality serves as the P-frame for inter prediction. By em- ploying weighted frames, the predictive frame approximates the reference frame more accurately, enhancing the overall performance of the prediction process. The experimental results, on a publicly available PET-CT dataset, demonstrate that the proposed prediction scheme outperformed the pre- viously proposed method, and attains coding gains up to 13.20% when compared with the single modality intra coding of the Versatile Video Coding (VVC) lossless standard.