My life-long goal is to build machines that can acquire common sense without human labeling and can be taught and instructed like humans: by observing how other humans complete tasks, and sometimes with the help of language.I believe common sense comes as a byproduct of using the right biases. For example, for visual understanding, humans have a strong bias on object permanence and most people understand scenes in terms of 3D. That is why we get suprised when a magician pulls a rabbit out of an empty hat. I have developed learning-based methods that are equipped with such biases and these models show strong generality on unseen situations. I have also designed algorithms for machines to learn simple concepts by watching human teachers doing things. Using the acquired knowledge, they are able to imagine their goal, self-supervise their own progress and efficiently pick up skills. I aim at designing learning algorithms that require only natural supervisions, e.g., self-supervision from predicting their raw sensory inputs (e.g., images and videos), supervision from human demonstrations, and priors learned from raw data.