Trans-oral robotic surgery (TORS) involves inserting a da Vinci surgical system into the mouth of a patient to dissect tumours in the back of the throat. Dr. Eitan Prisman is working with Wanwen Chen, a PhD student in my lab, and Randy Moore, a master's student in my lab, to facilitate TORS using augmented reality overlays of pre-operative imaging.
However, intraoperative imaging is more dynamic and can provide real-time feedback to the surgeon, letting them effectively see through tissue. Thus, we are working on integrating ultrasound on the outside of the neck. For this, I designed adapters for the da Vinci ProGrasp tool to hold an ultrasound device. However, we found that it would be easier to have a person (eg. the resident who is assisting) hold the probe in place. For this, the resident should be guided as to where on the patient to place the probe. This is closely related to my work on MR human teleoperation, so I created a HoloLens application to show the resident a virtual probe with which they align their real probe. This virtual probe can be controlled by the surgeon on the console, or automatically by keeping the surgical tools in the ultrasound plane. I therefore implemented precise HoloLens-based IR marker tracking to enable registration between HoloLens and robot, as well as a communication system and UI to send desired and measured poses and ultrasound images between the surgeon console and the HoloLens.