SurgAgentXR just won 2nd place in the AI × XR for Healthcare track at the XR Creator Con Hackathon 2026 finals in Berlin!
Together with Ajay Alageshan from the Visualisation Group at the University of Magdeburg and Ana Sodia Santos (external), Katarzyna Olszyńska and Prakash Pandi took part in the XRCC Hackathon in Berlin. Everything started with one question: surgical AI agents can reason, and AR navigation systems can render, but why can’t any system do both in the operator’s own spatial frame?
So they built SurgAgentXR: a surgical agent framework with eyes in the room. On a Meta Quest 3, a trainee performs an ultrasound-guided needle biopsy on a physical phantom with real probe, real needle, real hands. The agent sees the registered CT, the live ultrasound, and the tracked instruments in one spatial scene. It briefs the case, coaches in real time, and debriefs performance with metrics afterward. A coach that never leaves the room.
Built on the Project MONAI VLM Surgical Agent Framework, extended into the egocentric spatial frame with MedGemma for conversational reasoning, Qwen for visual queries, and a deterministic geometry layer so every number the agent speaks is computed, not hallucinated.
Grateful to immersive insiders (organizing XRCC) for organizing an amazing event, to Experimental Surgery | Digital Lab | Charité for bringing real clinical perspective to the track.
The agent isn’t more capable. It’s better situated. And this is just the beginning.