3D Reconstruction from Arthroscopic Images using NeRF: a preliminary in-silico study

Knee arthroscopy provides intra-articular visualization through a monocular camera and is routinely used in minimally invasive orthopedic surgery. During surgery, the acquired 2D arthroscopic images are essential for surgeons to visualize anatomical structures and treat cartilage lesions. Preoperative CT or MRI imaging is primarily used for diagnosis and patient-specific characterization. To effectively transfer this diagnostic information to the intraoperative setting and support patient-specific treatment, accurate registration between intraoperative arthroscopic images and preoperative 3D knee models derived from CT or MRI is required. Such registration relies on reconstructing a consistent 3D representation of the joint from arthroscopic images, a step that remains particularly challenging due to the limited field of view, partial surface visibility, and the absence of explicit depth information inherent to monocular arthroscopy.

Several approaches have been proposed to reconstruct 3D geometry from monocular images. Classical methods based on visual descriptors or SLAM (simultaneous localization and mapping), such as OneSLAM [1], achieve good performance in textured environments, but their accuracy significantly degrades in knee arthroscopy, where surfaces exhibit very little texture. More recent approaches rely on the joint estimation of depth and camera motion using deep learning models, including EndoSLAM [2], Endo-Depth-and-Motion [3], and EndoDAC [4]. While these methods have demonstrated strong potential for 3D reconstruction in endoscopy, their performance also decreases in arthroscopy due to the same visual challenges mentioned above.

Neural Radiance Fields (NeRF) [5] have recently emerged as a powerful approach for dense 3D reconstruction from monocular image sequences. The approach is based on learning a neural function that maps each 3D point and viewing direction to a volumetric density and color, enabling both the synthesis of consistent novel views and the inference of scene geometry. This representation is particularly well suited to monocular imaging setups, where explicit depth information is unavailable. Although initially introduced for natural scenes [6], NeRF-based methods have subsequently been adapted to minimally invasive surgery, notably through approaches such as MIS-NeRF [7], designed for laparoscopic images. These adaptations aim to address visual challenges inherent to laparoscopic imaging, such as variable illumination from the laparoscopic light source, partial surface visibility, and strong specular reflections. However, the application of NeRF-based reconstruction to knee arthroscopy remains largely unexplored.

In this work, we investigate the feasability of partial 3D reconstruction of the knee in the intraoperative setting from simulated monocular arthroscopic images, using MIS-NeRF as the reconstruction framework. Our objective is to assess the potential of this approach in the constrained context of knee arthroscopy.

We evaluate the intraoperative 3D reconstructions both qualitatively and quantitatively. Figure 2 presents a visual comparison between the ground-truth 2D images and the rendered images produced by MIS-NeRF for two representative patients’ knees.

For the quantitative evaluation, the reconstruction quality was assessed by averaging the results across the six datasets. The model achieved a PSNR of $31.88\pm 2.82$, an SSIM of $0.98\pm 0.004$, and an LPIPS of $0.017\pm 0.006$.

In this work, we investigated the feasibility of applying MIS-NeRF [7] to knee arthroscopy, a particularly challenging setting due to low-texture surfaces and constrained intra-articular viewpoints. The presented results demonstrate that MIS-NeRF can reconstruct a coherent 3D representation of the knee from simulated monocular arthroscopic image sequences.

The quantitative results show a PSNR of $31.88\pm 2.82$, indicating good pixel-level agreement between the rendered and reference images. An SSIM of $0.98\pm 0.004$ further confirms that structural information is well preserved. In addition, the low LPIPS value of $0.017\pm 0.006$ indicates a high perceptual similarity between the rendered and reference images, suggesting that the reconstructed appearance remains consistent despite the challenging visual characteristics of arthroscopic scenes. Overall, these results demonstrate that MIS-NeRF generalizes well to simulated knee arthroscopy, achieving reconstruction quality comparable to that reported for liver laparoscopy in the original MIS-NeRF [7] study.

Future work will focus on evaluating the geometric accuracy of the reconstructed models for computer-assisted arthroscopy by registering the reconstructed meshes to the corresponding preoperative anatomical models and measuring the Target Registration Error (TRE) using femoral anatomical landmarks. Finally, the proposed framework will be validated on real arthroscopic data to assess its clinical applicability.

This work was supported by funding from the French government through the National Research Agency (ANR), under grant numbers ANR-23-CPJ1-0131-01 and ANR-23-CE33-0007. The author would like to thank the authors of MIS-NeRF [7] for valuable discussions and encouragement. This work has been also supported by the Spanish Ministry of Science and Innovation MCIN/AEI/10.13039/501100011033 through project METAMORPH (PID2023-151295OB-I00)