Surgical User Interfaces



The Medical / Operating Room Interaction System (M/ORIS) program is one of the components of the Computer Aided and Image Guided Medical Interventions (CO-ME) program. The M/ORIS program is developing prototype approaches to non-contact interfaces in surgical environments and enables higher levels of interaction between surgeons and computerized medical equipment.


/webdav/site/vrai-group/shared/projects/nccr/images/caversaccio_small.jpgSurgical equipment and procedures rely increasingly on Information Technology (IT). Visualization and navigation tools used during surgery can provide extremely useful information to surgeons by the means of virtual reality, augmented reality. Allowing the surgeon to access information about the procedure and the patient in real-time during the intervention has the potential to make surgery less invasive, more accurate and faster, while reducing the costs.

Traditional Human-Computer Interface (HCI) tools (mouse, keyboard, etc.) are difficult to use in surgical environments because they cannot be placed within reach nor sterilized or simply because they are not appropriate for the nature of the interaction required. Moreover, the amount and complexity of information to be retrieved from the computer is significant and would add to the already heavy surgical workload, making the machine a hassle rather than an aid. Thus, there is a need for a high level, minimally invasive and sterile interaction tools that would allow surgeons to benefit from the computers skills with ease.

Computer Vision is a promising candidate for this purpose. It is compatible with all the physical requirements of a surgical environment (geometry, sterility, maintenance, etc.) and can be integrated with existing HCI technologies. Moreover, the controlled environment of an operating room ((OR), (controlled lighting, controlled obstructions, etc.) is particularly suitable for computer vision. We believe that Vision-Based Interfaces (VBI) can be accurate and reliable enough to be used in a demanding medical context.

Project objectives

/webdav/site/vrai-group/shared/projects/nccr/images/megad_chauncey.jpgM/ORIS will rely mostly on the input of a color, stereoscopic camera (see picture) to monitor the surgical procedure. In particular, M/ORIS will monitor the surgeon’s activity and integrate it with information available from the other surgical equipment. The system must fullfill the following objectives in real-time to successfully complete its role as a high-level interaction system:

Detection. Robust detection of features that are relevant for interaction (face, hands, arms, etc.) is a critical component of the system. The features must be found, identified and organized in a meaningful way.

Tracking. Once the features are detected, they must be tracked in space. The quality of the tracking must be constantly assessed, and a robust integration of tracking with detection is necessary to make the system as reliable as possible.

Activity monitoring. The output of the feature tracker must be matched with the procedure protocol, possibly using data from other sources as well. Depending on the state of the procedure, the computerized equipment in the OR is adjusted automatically to provide the necessary information or guidance.

Gesture interpretation. For some operations, direct gestural input from the user may be desirable. M/ORIS must correctly interpret command gestures, independently of the user.

Current work

Non-contact mouse. Based on the experience gained with HOT and the resulting TLIB image processing library, a demonstrator of a reliable interface to control a mouse cursor using hand gestures has been implemented and successfully tested by novice users.

Hand/arm detection and identification. Based on the experience gained with HOT and the resulting TLIB image processing library, a more advanced hand/arm detection algorithm is currently being developed using color and stereoscopic information.

Hand/arm tracking. Similarly, a tracking algorithm that integrates the detection method is being developed.

HMM gesture recognition. Using HMM to process stereoscopic, color information, a gesture pattern recognition engine is currently being developed. Different training algorithms and feature combinations are being evaluated to determine the most useful configuration for M/ORIS.


This project is one module of the activities lead by the EPFL-VRAI group within the PNR Co-Me project. Some parts of it have been achieved in collaboration with the Interaction Lab, UCSB (Santa Barbarba, USA).