Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation

PLOS ONE RESEARCH ARTICLE Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation Javier Pérez de Frutos ID1*, André Pedersen1,2,3, Egidijus Pelanis4, David Bouget1, Shanmugapriya Survarachakan5, Thomas Langø1,6, Ole-Jakob Elle4, Frank Lindseth5 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 1 Department of Health Research, SINTEF, Trondheim, Norway, 2 Department of Clinical and Molecular Medicine, Norwegian University of Technology (NTNU), Trondheim, Norway, 3 Clinic of Surgery, St. Olavs hospital, Trondheim University Hospital, Trondheim, Norway, 4 The Intervention Centre, Oslo University Hospital, Oslo, Norway, 5 Department of Computer Science, Norwegian University of Science and Technology (NTNU), Trondheim, Norway, 6 Research Department, Future Operating Room, St. Olavs hospital, Trondheim University Hospital, Trondheim, Norway * Abstract OPEN ACCESS Citation: Pérez de Frutos J, Pedersen A, Pelanis E, Bouget D, Survarachakan S, Langø T, et al. (2023) Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation. PLoS ONE 18(2): e0282110. https:// doi.org/10.1371/journal.pone.0282110 Editor: Paolo Cazzaniga, University of Bergamo: Universita degli Studi di Bergamo, ITALY Received: December 5, 2022 Accepted: February 8, 2023 Purpose This study aims to explore training strategies to improve convolutional neural networkbased image-to-image deformable registration for abdominal imaging. Methods Different training strategies, loss functions, and transfer learning schemes were considered. Furthermore, an augmentation layer which generates artificial training image pairs on-thefly was proposed, in addition to a loss layer that enables dynamic loss weighting. Published: February 24, 2023 Peer Review History: PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0282110 Copyright: © 2023 de Frutos et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: The IXI dataset is made available under the Creative Commons CC BY-SA 3.0 license, from https://brain-development. org/ixi-dataset/. The Oslo-CoMet dataset is not Results Guiding registration using segmentations in the training step proved beneficial for deeplearning-based image registration. Finetuning the pretrained model from the brain MRI dataset to the abdominal CT dataset further improved performance on the latter application, removing the need for a large dataset to yield satisfactory performance. Dynamic loss weighting also marginally improved performance, all without impacting inference runtime. Conclusion Using simple concepts, we improved the performance of a commonly used deep image registration architecture, VoxelMorph. In future work, our framework, DDMR, should be validated on different datasets to further assess its value. PLOS ONE | https://doi.org/10.1371/journal.pone.0282110 February 24, 2023 1 / 14 PLOS ONE publicly available as per agreement with the data holders. Access can be requested to The Intervention Centre, Oslo University Hospital, Oslo, Norway (contact via Dr. Ole-Jakob Elle, ), for researchers who meet the criteria for access to confidential data. Funding: This study was supported by the H2020MSCA-ITN Project No. 722068 HiPerNav; Norwegian National Advisory Unit for Ultrasound and Image-Guided Therapy (St. Olavs hospital, NTNU, SINTEF); SINTEF; St. Olavs hospital; and the Norwegian University of Science and Technology (NTNU). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. Learning deep abdominal CT registration Introduction For liver surgery, minimally invasive techniques such as laparoscopy have become as relevant as open surgery [1]. Among other benefits, laparoscopy has shown to yield higher quality of life, shorten recovery time, lessen patient trauma, and reduce blood loss with comparable long-term oncological outcomes [1]. Overcoming challenges from limited field of view to manoeuvrability, and a small work space are the foundations of laparoscopy success. Imageguided navigation platforms aim to ease the burden off the surgeon, by bringing better visualisation techniques to the operating room [2, 3]. Image-to-patient and image-to-image registration techniques (hereafter image registration) are at the core of such platforms to provide clinically valuable visualisation tools. The concept of image registration refers to the alignment of at least two images, matching the location of corresponding features across images in order to express them into a common space. Both rigid and non-rigid registration are the two main strategies to define the alignment between the images. Rigid registration uses affine transformations, which are quicker to compute but less accurate as these are applied globally. Nonrigid registration, also known as deformable registration, defines a diffeomorphism, i.e., a point-to-point correspondence, between the images. However, non-rigid registration comes at the expense of higher computational needs and thus hardware constraints might hinder the development and deployment of such algorithms. In medicine, image registration is mandatory for fusing clinically relevant information across images; groundwork for enabling imageguided navigation during laparoscopic interventions [4, 5]. Additionally, laparoscopic preoperative surgical planning benefits from abdominal computed tomography (CT) to magnetic resonance imaging (MRI) registration to better identify risk areas in a patient’s anatomy [6]. During laparoscopic liver surgeries, intraoperative imaging (e.g., video and ultrasound) is routinely used to assist the surgeon in navigating the liver while identifying the location of landmarks. In parenchyma-sparing liver resection (i.e., wedge resection) for colorectal liver metastasis, a minimal safety margin around the lesions is defined to ensure no recurrence and spare healthy tissue [7]. When dealing with narrow margins and close proximity to critical structures, a high accuracy in the registration method employed is paramount to ensure the best patient outcome. Patient-specific cross-modality registration between images of different nature (e.g., CT to MRI) is practised [8], yet being a more complex process compared to mono-modal registration. The alignment of images can be evaluated through different metrics based either on intensity information (...truncated)