From CT Scan to 3D Model: How AI Bone Segmentation Is Changing Surgical Planning
How deep learning transforms CT scans into patient-specific 3D bone models in minutes — replacing hours of manual segmentation and enabling precision surgical planning for knee osteotomy, arthroplasty, and complex reconstruction.
The Bottleneck in 3D Surgical Planning
Every surgeon who has used 3D preoperative planning knows the bottleneck: segmentation. Before a CT scan becomes a usable 3D model, someone must trace the boundaries of each bone — slice by slice, across hundreds of images — separating femur from tibia, patella from fibula, cortical bone from soft tissue.
For a single knee CT, manual segmentation takes 2–4 hours of skilled technician time. For complex cases involving multiple bones or fracture fragments, the time can exceed a full working day. This time cost has historically limited 3D planning to complex cases where the investment was clearly justified — revision arthroplasty, tumour resection, severe deformity correction — leaving routine cases with conventional 2D planning.
Deep learning has fundamentally changed this equation. AI-powered bone segmentation can now process a knee CT and produce individual 3D models of the femur, tibia, patella, and fibula in minutes rather than hours, with accuracy approaching expert manual segmentation.
How AI Bone Segmentation Works
The Pipeline: CT to 3D Model
The journey from CT scan to surgical 3D model follows a consistent pipeline, whether performed manually or with AI assistance:
Acquisition: A standard knee CT protocol produces a series of axial DICOM images — typically 200–500 slices at 0.5–1.0mm spacing. The raw data contains full volumetric information about bone density, soft tissue, and anatomical geometry.
Preprocessing: The CT volume is normalised — adjusting Hounsfield unit windowing to optimise bone-soft tissue contrast, resampling to a consistent voxel spacing, and cropping to the region of interest. This step is critical: AI models are sensitive to input data characteristics, and preprocessing ensures the model receives data in the format it was trained on.
Segmentation: Each voxel (3D pixel) in the CT volume is classified as belonging to a specific structure — femur, tibia, patella, fibula, or background. This is where AI replaces manual effort: a deep learning model processes the entire volume and produces a multi-label segmentation mask in a single forward pass.
Mesh generation: The segmentation mask is converted to 3D surface models using marching cubes or similar algorithms. Each labelled bone becomes an individual STL (Standard Tessellation Language) mesh — the standard format for 3D printing and surgical planning software.
Post-processing: The raw meshes are smoothed (removing staircase artefacts from the voxel grid), decimated (reducing polygon count for interactive performance), and quality-checked for anatomical completeness.
The Models: From U-Net to nnU-Net
Medical image segmentation has converged on variants of the U-Net architecture, originally developed for 2D biomedical image segmentation and subsequently extended to 3D.
U-Net uses an encoder-decoder structure with skip connections. The encoder progressively downsamples the input to capture context; the decoder upsamples back to the original resolution using the skip connections to preserve fine spatial detail. While U-Net was designed for 2D data, 3D U-Net extends this to process entire volumes rather than individual slices — critical for maintaining anatomical continuity across the z-axis.
nnU-Net (Isensee et al., 2021) has become the de facto standard for medical image segmentation benchmarks. Rather than a single model, nnU-Net is a self-configuring framework that automatically adapts preprocessing, architecture, and training parameters to each dataset. It consistently achieves state-of-the-art or near-state-of-the-art performance across diverse segmentation tasks without manual hyperparameter tuning.
TotalSegmentator (Wasserthal et al., 2023) extended this approach to whole-body CT segmentation, training on over 1,000 annotated CT scans to segment 104 anatomical structures — including all major bones, organs, and vessels. For orthopaedic applications, TotalSegmentator provides a robust starting point that can be fine-tuned for specific anatomical regions.
What Can Be Segmented?
Knee: The Most Developed Application
Knee CT segmentation is the most mature orthopaedic application. A well-trained model produces four individual bone models from a single knee CT:
- Femur — distal femoral anatomy, condylar geometry, mechanical axis
- Tibia — tibial plateau topology, posterior slope, proximal anatomy
- Patella — patellar tracking assessment, thickness measurement
- Fibula — proximal fibula anatomy, relevant for posterolateral corner procedures
Published Dice similarity coefficients for knee bone segmentation typically exceed 0.95 for femur and tibia, 0.85–0.90 for patella, and 0.80–0.85 for fibula. The lower scores for smaller bones reflect their smaller volume (where even small errors disproportionately affect the metric) rather than clinically significant inaccuracy.
Hip: Arthroplasty and Deformity
Hip CT segmentation enables automated measurement of acetabular version, femoral anteversion, offset, and leg length — parameters critical for total hip arthroplasty planning. AI-based planning has been shown to increase the acetabular component fit from approximately 30–57% with manual planning to 66–90% with AI assistance.
Spine: The Emerging Frontier
Vertebral segmentation from CT has advanced significantly, with models achieving Dice coefficients above 0.90 for individual vertebral body segmentation. Applications include pedicle screw trajectory planning, spinal deformity assessment, and tumour resection planning.
Fracture Fragments: The Hardest Problem
Segmenting fracture fragments — irregular bone pieces displaced from their anatomical position — remains the most challenging task. Fragments may be small, overlapping in the CT volume, or obscured by metallic artefacts from prior fixation. Specialised models are emerging for specific fracture patterns, but generalised fracture segmentation remains an active research area.
From Segmentation to Surgical Planning
The 3D bone model is the foundation; the clinical value comes from what the planning software enables the surgeon to do with it.
Mechanical Axis Analysis
Accurate bone models enable automated measurement of mechanical and anatomical axes — HKA, MPTA, LDFA, posterior tibial slope — from the 3D geometry rather than 2D projection. This eliminates rotational errors inherent in radiographic measurement and provides true 3D alignment assessment.
Patient-Specific Instrumentation (PSI)
3D bone models feed directly into CAD workflows for designing patient-specific surgical guides. Our published research in OJSM demonstrated that PSI guides reduce alignment variability across different surgeon experience levels.
Virtual Osteotomy Simulation
Before cutting bone, the surgeon can simulate the planned osteotomy on the 3D model — visualising the correction angle, assessing hinge point placement, and confirming that the planned correction achieves the desired mechanical axis. This is particularly valuable for HTO and DFO planning, where small angular differences significantly impact clinical outcome.
Implant Templating
Accurate bone models enable automated implant sizing and positioning. Machine learning models trained on surgical registry data can predict optimal component size from the 3D geometry, reducing intraoperative trial-and-error and enabling just-in-time implant logistics.
Performance: How Good Is AI Segmentation?
The honest answer depends on what is being segmented and how accuracy is measured.
Large bones (femur, tibia): Dice coefficients consistently exceed 0.95 — effectively indistinguishable from expert manual segmentation for clinical planning purposes. Surface distance errors are typically below 1mm.
Small bones (patella, fibula): Dice coefficients range from 0.80–0.90. While numerically lower, this reflects the mathematical sensitivity of Dice to small volumes. Clinically, the segmentation is usually adequate for planning but should be visually verified.
Articular surfaces: The joint surface — where surgical precision matters most — is where segmentation quality must be highest. Models specifically trained with articular surface annotations achieve sub-millimetre surface distance accuracy, but this requires dedicated training data.
Pathological anatomy: Severe osteoarthritis (large osteophytes, subchondral cysts), prior metallic implants, and fracture deformity all challenge segmentation models. Performance degrades in these cases, and manual correction becomes more likely.
Explainability in Segmentation
Visual explanations and confidence metrics are as important for segmentation models as they are for classification models. A segmentation tool that produces a 3D model without any indication of confidence leaves the surgeon unable to assess where manual verification is needed.
Confidence maps — heatmaps showing voxel-level model uncertainty — can highlight regions where the segmentation is less reliable. Joint margins, thin cortical bone, and pathological anatomy typically show higher uncertainty.
The Architecture Question: Where Should Segmentation Run?
CT bone segmentation requires significant computational resources — GPU inference for a single knee CT takes 30 seconds to 2 minutes on a modern GPU, or 2–5 minutes on CPU. This creates an architectural decision for clinical software.
Server-side processing sends the CT volume to a GPU-equipped server for segmentation. This provides fast inference but requires transmitting patient imaging data to an external server — creating KVKK, HIPAA, and GDPR compliance obligations.
Client-side processing runs inference in the clinician's browser. For lightweight models (OA grading from single radiographs), this works well with ONNX Runtime Web. For heavy 3D segmentation models, browser-based inference is not yet practical — the model size (hundreds of MB) and computational requirements exceed current browser capabilities.
Hybrid processing offers a middle path: CT data is uploaded to a secure processing server, segmentation runs server-side, and only the resulting STL meshes (which contain no patient-identifiable information) are returned to the client. The original CT data is deleted after processing. This preserves the privacy advantage for the final clinical interaction while enabling GPU-accelerated segmentation.
Clinical Validation: What the Evidence Shows
For knee osteotomy planning, 3D planning with AI-segmented models has been shown to improve the precision of angular correction by up to 44.7% compared to conventional 2D planning — a clinically meaningful improvement in a procedure where one degree of mechanical axis deviation can significantly impact long-term outcomes.
External validation across multiple institutions and imaging equipment is essential and ongoing. Models trained on one scanner manufacturer's CT protocol may underperform on images from a different manufacturer — a challenge that multi-institutional training datasets address but do not eliminate.
What Should Surgeons Know?
For orthopaedic surgeons considering AI-powered 3D planning, five practical considerations apply:
Segmentation is not planning. A 3D model is the starting point, not the endpoint. The clinical value depends on what the planning software enables the surgeon to do with the model — osteotomy simulation, implant templating, PSI design, mechanical axis analysis.
Verify small bones and pathological anatomy. Large bone segmentation is reliable. Small bone segmentation (patella, fibula, fracture fragments) should be visually verified and may require manual correction.
Ask about training data. A model trained on 50 CTs from one institution will perform differently than one trained on 500 CTs from ten institutions. Dataset size, diversity, and annotation quality directly impact clinical reliability.
Understand the privacy architecture. Know whether your CT data is processed locally, on an external server, or through a hybrid pipeline. This determines your data protection obligations.
Demand editability. Any AI segmentation tool should allow the surgeon to manually correct the output. Automated segmentation that cannot be verified and adjusted is not clinically acceptable.
Salnus's Approach
Salnus is developing AI-powered bone segmentation as part of our preoperative planning platform. Our current pipeline segments four knee bones (femur, tibia, patella, fibula) from standard CT DICOM data, generating individual STL meshes for 3D visualisation and surgical planning.
We are training custom nnU-Net models on 167 annotated knee CT volumes with expert-verified segmentation labels. Our hybrid processing architecture uses server-side GPU inference for segmentation with client-side 3D visualisation — patient CT data is processed transiently and deleted after mesh generation.
The platform is currently designated as Research Use Only, with development toward CE marking underway.
For surgeons interested in evaluating our segmentation pipeline or contributing annotated CT data for model training, contact our team.
Disclaimer: This article is for educational and research purposes only. Salnus tools are designated for Research Use Only (RUO) and are not cleared medical devices. Clinical decisions should be made by qualified physicians based on comprehensive patient assessment.
References:
- Isensee F, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods. 2021;18:203-211.
- Wasserthal J, et al. TotalSegmentator: Robust Segmentation of 104 Anatomical Structures in CT Images. Radiology: AI. 2023;5(5):e230024.
- Han Y, et al. Artificial Intelligence in Orthopedic Surgery: Current Applications, Challenges, and Future Directions. MedComm. 2025.
- Zhou X, et al. 3D preoperative planning improves precision of orthopaedic surgeries. Int J Surg. 2023.
- Salnus Research Group. 3D-Printed Patient-Specific Guides for Knee Reconstruction. OJSM. 2026.
Reviewed by the Salnus biomedical engineering team.