Previous work from the MEDCVR lab has shown strong results in policy learning for surgical robotic manipulation tasks using methods such as Reinforcement Learning (RL) and Imitation Learning (IL), but most approaches rely on single camera 2D images. However, as the task becomes more complex: the robot must interact in a 3D world requiring spatial reasoning and depth understanding and multi-object interaction, on which single-view learning does not scale effectively. This project will develop methods to capture consistent 3D representations across viewpoints. These representations can then be used to train policies that are more robust to camera viewpoints, and capable of handling inherently 3D tasks such as surgical cutting.  This project aims to: Build on previously developed representation learning pipeline;  Train policies for 3D manipulation tasks using multiple cameras in simulation; Transfer trained policies to real robots to evaluate performance on real 3D tasks; Compare baseline single-view RL methods to quantify improvements in sample efficiency and robustness to multiple camera viewpoints; and, Publish the findings.

 

The SUDS Scholar will work closely with the lab’s Amey Pore, Schmidt AI in Science Postdoctoral Fellow. The SUDS Scholar responsibilities will include: