Using Simulation to Enable Real World Robotics

Candidacy Exam Paper List
David Watkins-Valls December 2018


Real world robotics is a multifarious process spanning several fields including simulation, semantic/scene understanding, reinforcement learning, domain randomization, just to name a few. Ideally simulators would accurately capture the real world perfectly in a much faster capacity allowing for a predictive power of how a robot will interact with its environment. Unfortunately, simulators neither have the speed nor accuracy to support this. Simulators, such as Gazebo, Webots, and OpenRave, are supplemented with machine learned models of their environment to solve specific tasks such as scene understanding and path planning. This can be compared to a physical only solution which can be costly in terms of price and time. Advances in virtual reality allow for new ways for humans to provide training data for robotic systems in simulation. Using modern datasets such as SUNCG and Matterport3D we now have more ability than ever to train robots in virtual environments. Through understanding modern applications of simulations, better robotic platforms can be designed to solve some of the most pressing challenges of modern robotics.


First I will discuss the background and motivation of simulation in robotics, citing a foundational paper discussing the nature of simulating motion elements in 1983. Then I will compare virtual simulation to the drawbacks of physical simulation. I will then do an in depth analysis of different simulation architectures as well as applications built on top of their backends. There are multiple simulation datasets in use today and I will mention how they help supplement current research. I will then discuss the human in the loop data that can be generated because of the ease of these simulators through virtual reality applications. I will then discuss how we are using these sophisticated datasets to train scene understanding network architectures and enable sim-to-real applications. Finally I will discuss the impacts and boons that simulation has given to deep learning and reinforcement learning applications.




Simulation Datasets

Virtual Reality

Scene Understanding


Machine Learning