We present an algorithm to generate diverse foreground objects and composite them into background images using a GAN architecture. Given an object class, a user-provided bounding box, and a background image, we first use a mask generator to create an object shape, and then use a texture generator to fill the mask such that the texture integrates with the background. By separating the problem of object insertion into these two stages, we show that our model allows us to improve the realism of diverse object generation that also agrees with the provided background image. Our results on the challenging COCO dataset show improved overall quality and diversity compared to state-of-the-art object insertion approaches.

Generative Adversarial Networks have shown promise in unpaired image translation. However, translating unpaired objects from unaligned domains is an unsolved problem. Existing methods are restricted to domain pairs which require only minor shape change, and they typically mode collapse on more challenging domain pairs such as giraffe to sheep, or racket to bat. First, with a simple toy dataset, we show that existing image translation models are inherently unequipped to handle significant shape change because they fail to disentangle the foreground and background. We present a novel method and network architecture to address this issue. The effectiveness of our network is confirmed by a user study, where we are preferred 2× over the SOTA. However, even our improved baseline suffers from instability on these diverse domains. Thus, we explore various multi-scale architectures. Additionally, we improve the efficiency of our networks because previous methods are highly resource intensive and require long training times. We then design a transformer-based network that is not limited by receptive field, as convolutions are. This enables it to make significant global changes to the object shape. We separate the shape and color translation problems so that we can utilize transformer-based models. Effectively, we translate shapes as if they were sentences. We find this enhances the quality and stability of the shape translations and explore ways our transformer representations can be combined with color networks.

Pose tracking is an important problem that requires identifying unique human pose-instances and matching them temporally across different frames of a video. However, existing pose tracking methods are unable to accurately model temporal relationships and require significant computation, often computing the tracks offline. We present an efficient Multi-person Pose Tracking method, KeyTrack, that only relies on keypoint information without using any RGB or optical flow information to track human keypoints in real-time. Keypoints are tracked using our Pose Entailment method, in which, first, a pair of pose estimates is sampled from different frames in a video and tokenized. Then, a Transformer-based network makes a binary classification as to whether one pose temporally follows another. Furthermore, we improve our top-down pose estimation method with a novel, parameter-free, keypoint refinement technique that improves the keypoint estimates used during the Pose Entailment step. We achieve state-of-the-art results on the PoseTrack'17 and the PoseTrack'18 benchmarks while using only a fraction of the computation required by most other methods for computing the tracking information.