The need for labeled data is one of the largest bottlenecks in training supervised learning models like deep neural networks. This is especially the case for many real-world tasks where large scale annotated examples are either too expensive to acquire or unavailable due to privacy or data access constraints. Many of these tasks involve users and as such allow access to a rich set of user interactions with the system (e.g. utterances, clicks in an intelligent assistant, activities performed within a productivity software, etc.). In this project, we develop techniques to leverage rich user interactions as a source of weak supervision to mitigate the scarcity of annotated examples, and develop robust deep neural network models for real-world applications.