Katarzyna Olszyńska and Robin Urrutia Escobar have bin to the 6th Summer School on AI, Robotics and XR Medical Augmented Reality Summer School (MARSS) for Clinical Applications in Shenzhen & Hong Kong. This was a great opportunity to dive deeper into the topics they work on daily – robotics, AI, and XR in the medical field.
Part of the Summer School was a hackaton where Katarzyna team managed to win the first place. Her team built an “Embodied AI-Driven Ultrasound Robotic System.” We developed an Intelligent AI Core powered by LLM/VLM with two key capabilities – procedural understanding, where the model comprehends ultrasound workflows and recognizes scanning stages in real time, and embodied control, where the LLM/VLLM follows human commands and interprets the live ultrasound images from the probe to guide its manipulation.
Robins team worked on “XR-Guided Robotic Ultrasound with CBCT Integration for Intervention.” The project focused on supporting accurate needle placement in spine interventions by combining robotic ultrasound, Cone-Beam CT data, and XR visualization. Our goal was to co-register live robotic ultrasound with 3D CBCT data, stream ultrasound information into an XR environment, and render a virtual spine phantom to support more intuitive 3D trajectory planning.
The experience was extremely rewarding for both of them. It was a great exercise in interdisciplinary collaboration, rapid prototyping, and translating technical ideas into clinically motivated concepts within a short time frame.
Thanks to the organizers – Prof. Dr. Hongbin Liu, Prof. Dr. Nassir Navab, Prof. Dr. Shaohua Kevin Zhou, and Prof. Dr. Jason Chan Ying for putting this together.