I professionally developed VR/AR surgery simulations. Many of these simulations use a haptic pen which allows surgeons to feel objects in 3D space. The goal of these simulations is to allow doctors a chance to practice surgeries in virtual space before the actual surgery occurs.
As Technical Designer, I develop Virtual Reality simulations that allow doctors to interact with real patient data in 3D space. I was responsible for the design and development of the simulations. Additionally I managed the User Experience and created videos to be used for marketing content. Below are a few of the simulations that I was in charge of developing.
The goal of the cranial vault surgery is for a doctor to rearrange the patient’s skull to prevent brain damage. This simulation provides a rendered version of a patient’s skull, broken up into four pieces, which the doctor manipulates when performing the procedure. On the monitors in the back of the operating room, there are images and information about the patient. Often when demoing this simulation, I would swap in the logo on the back monitor for the visiting company to add some personalization. There is also a near monitor that contains the operation steps. Finally the closest monitor is for user interaction with the tools I built into the simulation.
Opacity:Allows the doctor to adjust the transparency of the skull pieces. This allows for more precision when seeing how pieces overlap with each other.
Measure:A ruler that allows doctors to know the distance between two points in the 3D space.
Draw:Allows the doctor to write down notes or draw whatever else may be needed at that point in 3D space.
Link:Connects the two large skull pieces together. This was a specific request from the doctor we were working with.
The goal of this surgery is to cut a hole in the skull that makes it easy for the doctor to access a tumor. The right monitor contains the steps in order to complete the surgery. The middle monitor contains CT images of the patient’s head that can be scrubbed through by using the Virtual Reality (VR) controller. The left monitor is where our tools are located and can be moved freely throughout the operating room. This three monitor setup: tool monitor, operation steps, and patient information is the standard set up I used across all of our simulations. If a doctor completed one of our simulations, he/she would already know the basics for our other simulations.
To start this simulation, we use a haptic pen to place three points on the patient’s skull which decide where a clamp will be placed to keep the head steady. From there, the surgeon can draw an outline of the part of the head they want removed. The left monitor allows the surgeon to undo or clear drawings if needed. When ready, the surgeon selects a point they want to be the center for removal and then I run a flood algorithm that takes all of the voxels that make up the skull up until the drawing and deletes them. To keep this as simple as possible, I designed most interactions to occur with the button press of the haptic pen.