A surgeon wearing a mixed-reality headset may soon be able to see a patient’s hidden anatomy floating directly over the operating field—tumors, blood vessels, bones and planned instrument paths appearing as interactive three-dimensional holograms. A major review of human surgical procedures suggests that this technology is moving beyond science-fiction imagery and into real operating rooms, although it remains far from becoming a routine replacement for conventional navigation systems. The review, published in 3D Printing in Medicine, examined 41 original studies involving 760 patients across a wide range of surgical specialties. Taken together, the evidence indicates that mixed reality can improve visualization, procedural precision and workflow in selected operations, but also exposes persistent problems involving registration errors, spatial drift, limited fields of view, eye fatigue and the reliability of holographic overlays.
Mixed reality is often confused with augmented reality and virtual reality, but the distinction matters in surgery. Virtual reality blocks out the physical environment and replaces it with a digital one, making it unsuitable for most live procedures. Augmented reality places digital information over the real world, but may offer little interaction with the virtual objects. Mixed reality attempts to anchor three-dimensional digital models within the physical environment so that the surgeon can move around them, resize them, rotate them or inspect them while still seeing the patient and the operating room. In practice, the technology links patient-specific scans with the surgeon’s real-time view through a head-mounted display, creating an interactive layer between medical imaging and direct observation.
The review classified intraoperative mixed-reality systems into three main groups. Mixed-reality surgical navigation tracks the position and orientation of instruments and relates them to a preoperative three-dimensional model, allowing the system to display a planned trajectory. Image guidance overlays a reconstructed anatomical structure onto the patient, but does not track the instruments themselves. Image reference is the simplest approach: a static or movable holographic model is placed beside the surgical field as a three-dimensional visual reference. The most sophisticated systems combine a headset with optical or electromagnetic tracking, while simpler platforms rely on the surgeon’s visual interpretation of an untracked model. This classification is important because the word “mixed reality” can describe systems with very different levels of technical capability and clinical risk.
Every system begins with medical imaging, usually computed tomography or magnetic resonance imaging stored in the DICOM format. Before a hologram can appear in the operating room, the relevant structures must be isolated from hundreds of two-dimensional image slices. This process, called segmentation, separates organs, bones, vessels, tumors or other regions of interest from surrounding tissue. Software then converts those segmented structures into a polygonal three-dimensional model that can be displayed by the headset. Some platforms use artificial intelligence to automate segmentation, often with convolutional neural networks or other deep-learning models. These algorithms can reduce the time required to outline anatomy and decrease variability between users, but the resulting models still require clinical verification because an inaccurate segmentation can produce a dangerously convincing hologram.
Registration is the technical step that determines whether the virtual anatomy actually matches the patient. The system must align corresponding points in the digital model and the physical body, using landmarks such as the nose, ear or bone surfaces, or using markers placed on the patient. Optical systems use infrared cameras to identify reflective markers, whereas electromagnetic systems use sensors to locate instruments within an electromagnetic field. In integrated platforms, an external navigation system may provide the instrument position while the headset supplies the surgeon’s heads-up visualization. The review reports that such integrated systems can approach the accuracy of established navigation technology. By contrast, stand-alone headsets that rely on inside-out tracking and simultaneous localization and mapping can show larger registration errors and occasional hologram drift as the surgeon moves around the patient.
The clearest clinical signals came from spinal surgery. Several studies examined mixed-reality guidance for placing pedicle screws, which must pass through narrow bony corridors without damaging the spinal cord or nearby nerves. In one series, 205 screws were placed with 98 percent classified as accurate. Another study involving nine patients reported that all screws met the accepted Gertzbein-Robbins grade A or B criteria, with 96.8 percent receiving the highest grade. In a randomized study of lumbar procedures, mixed reality was associated with less bleeding, shorter operating times, fewer fluoroscopic images and a higher first-pass success rate. Other reports described reduced radiation exposure during minimally invasive spinal puncture and improved targeting of vertebral fractures. These results are promising, but most studies were small, and highly experienced teams often performed the procedures, limiting how easily the findings can be generalized.
Neurosurgery offered another striking example of the technology’s potential. In external ventricular drain placement, a catheter must be inserted into the brain along a trajectory calculated from preoperative scans. A holographic guide reduced the average number of insertion passes from 2.33 with the conventional freehand method to 1.07, without reported adverse events in the study. Mixed reality was also used to plan cranial incisions, guide puncture and drainage of intracerebral hematomas, and visualize brain lesions during tumor surgery. In a series of 37 intracranial procedures, 81.1 percent of lesions showed less than 5 millimeters of deviation from conventional neuronavigation. However, the additional preparation time averaged more than half an hour, and stand-alone holographic systems remained less accurate than established navigation in some comparisons.
The applications extended well beyond the brain and spine. In oral and maxillofacial tumor surgery, mixed-reality navigation produced an average deviation of 1.68 millimeters between planned and actual bone cuts, and all patients had negative resection margins in the reported series. In urology, a study of 100 laparoscopic nephrectomy patients found that the mixed-reality group had shorter operations, shorter warm-ischemia times and less blood loss; successful laparoscopic completion occurred in 82 percent of assisted cases compared with 46 percent of non-assisted cases. In breast-cancer surgery, holographic tumor localization was successfully matched with conventional carbon markings, while a 300-patient sentinel lymph-node study reported faster detection and fewer complications when mixed reality supplemented dye-based localization. Other studies used three-dimensional vascular maps to identify perforating vessels for reconstructive flaps, locate non-palpable lung nodules and guide pulmonary interventions.
In complex abdominal operations, the technology is particularly useful as a spatial memory aid. Surgeons have used patient-specific holograms to visualize liver tumors, portal veins, hepatic arteries and planned dissection planes, including in patients with numerous colorectal metastases or unusual hilar anatomy. During a minimally invasive liver procedure, a hologram helped guide a puncture for injecting indocyanine green, a fluorescent dye used to demarcate tissue under near-infrared imaging. In ophthalmic surgery, surgeons treating severe tear-duct obstruction could rotate, slice and inspect three-dimensional models of distorted anatomy during endoscopic dacryocystorhinostomy. The model could be displayed beside the surgical field without blocking the endoscope, offering a way to revisit hidden structures as the operation progressed. These examples show that mixed reality does not always need to steer an instrument directly; sometimes its greatest value is helping the surgeon understand anatomy that cannot be seen from the surface.
The dominant hardware in the reviewed literature was Microsoft’s HoloLens, used in 26 of the 41 studies, followed by HoloLens 2 in nine. Magic Leap 2 and dedicated navigation systems such as the xvision Spine System also appeared. Most surgical applications used “see-through” displays, in which transparent lenses allow the surgeon to view the real patient while holographic objects are projected into the field of vision. Pass-through headsets instead use cameras to capture the room and display that video on an opaque internal screen. Although pass-through systems can render virtual objects effectively, their representation of the physical world may be affected by camera resolution, lighting and high-contrast conditions common in operating rooms. See-through devices therefore remain more practical for many procedures, despite their own limitations.
Those limitations are substantial. Surgeons have reported eye fatigue, physical discomfort and, in some cases, cybersickness. Narrow fields of view can force users to move their heads to keep critical information visible, while latency, rendering artifacts and wireless delays can make a hologram feel disconnected from the patient. Voice commands and hand gestures are not always reliable, particularly when the operating team is wearing gloves or working in a noisy, crowded theatre. Bright surgical lights can compete with holographic images, and focal rivalry can make it difficult for the eyes to focus simultaneously on a nearby virtual object and the real anatomy. More fundamentally, soft organs deform when manipulated, move with respiration and shift relative to preoperative scans. A perfectly registered liver model can therefore become imperfect within minutes of the first incision.
Spatial drift creates a related danger. Inside-out tracking systems construct a map of the environment and use it to maintain the hologram’s position, but small errors can accumulate as the user walks around the patient, lighting changes or equipment is moved. One clinical study observed approximately 1–2 millimeters of drift, while other stand-alone systems reported registration errors substantially greater than those of conventional navigation. Researchers are testing vision-based relocalization, fiducial markers, periodic re-registration and fusion with external trackers to stabilize the overlay. These safeguards may make integrated mixed-reality systems more dependable, but they also add equipment, preparation and cost. In some reported workflows, creating the three-dimensional model required two to three hours, with another 20–30 minutes needed in the operating room.
The review’s authors describe mixed reality as an emerging technology rather than an established standard of care. Only a small number of the systems studied were integrated, regulated platforms; many relied on commercially available headsets adapted for experimental clinical use. The evidence base was also heterogeneous, combining case reports, small case series, feasibility studies and a limited number of controlled trials. That makes it difficult to determine whether better visual guidance consistently improves long-term patient outcomes, rather than simply making procedures feel more intuitive to surgeons. Larger randomized studies will be needed to measure complications, revision rates, radiation exposure, operating time and cost across hospitals with different levels of technical expertise.
Artificial intelligence may determine how quickly the technology matures. Automated segmentation could turn hours of manual image processing into a faster, more reproducible workflow, while future systems may combine imaging, instrument tracking and real-time anatomical updates. A headset that recognizes a surgical instrument, predicts its trajectory and continuously adjusts a model as tissue moves would be far more powerful than a static hologram. Yet those systems would also raise higher demands for validation, cybersecurity, explainability and regulatory oversight. For now, mixed reality’s most credible role is as an additional layer of information—not an autonomous surgeon and not a substitute for clinical judgment. The review suggests that holographic guidance is already useful in selected operations, but its viral appeal should not obscure the central medical question: whether the technology can reliably make surgery safer for patients.
Subject of Research: Intraoperative mixed-reality applications in human surgery across multiple surgical disciplines
Article Title: Intraoperative use of mixed reality (MR) in humans across surgical disciplines: a major review
Article References: Nowak R, Nowak M, Rękas M, et al. “Intraoperative use of mixed reality (MR) in humans across surgical disciplines: a major review.” 3D Printing in Medicine 11, 59 (2025). Original research article
Image Credits: AI Generated
DOI: 10.1186/s41205-025-00305-7
Keywords: mixed reality surgery, surgical navigation, holographic guidance, intraoperative imaging, augmented reality, three-dimensional medical imaging, artificial intelligence segmentation, head-mounted displays, neurosurgery, spinal surgery
Tags: 3D hologram in surgeryaugmented reality vs mixed reality in medicinechallenges of mixed reality in surgeryfuture of mixed reality in surgical practiceholographic overlays for surgerymedical visualization technologymixed reality eye fatigue and usability issuesmixed reality in minimally invasive proceduresMixed reality surgical applicationsmixed reality surgical navigationmixed reality workflow improvements in healthcarespatial registration errors in medical mixed reality
