New hybrid approach for AR visualization of ultrasound images has surgical use

Children’s National Hospital researchers combined augmented reality (AR) and ultrasound imaging technologies to improve visualization during laparoscopic procedures. The patent-pending hybrid tracking method improved accuracy compared with hardware-based and computer vision-based approaches, according to the pre-clinical study published in the Journal of Medical Imaging.

“The system that we developed can work reliably under challenging intraoperative conditions, which is exciting for us,” said Raj Shekhar, M.S., Ph.D., principal investigator at Children’s National and senior author in the study. “The surgical view is usually very complex with various tissues, tools, blood and smoke in the view. Our system has been evaluated on pre-clinical models and has shown to provide acceptable overlay accuracy even during extreme conditions.”

Augmented reality for surgery is an emerging trend, consisting of a computer-generated image for the operating surgeon to see. While using ultrasound during surgery allows the doctor to see the organ’s internal structures, when combined, the AR system significantly improves the surgeon’s comprehension of the ultrasound image in the context of the surgical view.

“The AR system that we developed will benefit patients because AR-guided surgeries can be more precise, safer and faster,” said Shekhar. “Through sustained research and development effort and robust clinical-technical partnership, Children’s National is leading in clinical translation and technology transfer of the laparoscopic AR visualization technology. Other solutions are more laboratory-based while our focus has been on building a clinically practical and viable solution from day one.”

Laparoscopic surgery is known to improve outcomes, cause less scarring and speed the recovery process since the surgeons create small incisions with the aid of a camera. Still, doctors have a limited view of anatomic structures and surgical targets below the exposed surfaces.

Given that AR is an emerging technology, this work would not have been possible until recently. According to the researchers, no other scientists in the field take advantage of both tracking hardware and computer vision-based approaches.

“This work is important because it features best of both worlds: relying on tracking hardware to ensure robustness while using computer vision to enhance accuracy. This will generate a more accurate and reliable AR system which is more feasible for clinical use,” said Shekhar.