In this work, we propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation. Unlike training a variational autoencoder (VAE) from scratch, which primarily emphasizes low-level details, our approach leverages the rich semantic structure of foundation encoders. We introduce a three-stage alignment strategy: (1) freeze the encoder and train an adapter and a decoder to establish a semantic latent space; (2) jointly optimize all components with an additional semantic preservation loss, enabling the encoder to capture perceptual details while retaining high-level semantics; and (3) refine the decoder for improved reconstruction quality. This alignment yields semantically rich image tokenizers that benefit diffusion models. On ImageNet 256$\times$256, our tokenizer accelerates the convergence of diffusion models, reaching a gFID of 1.90 within just 64 epochs, and improves generation both with and without classifier-free guidance. Scaling to LAION, a 2B-parameter text-to-image model trained with our tokenizer consistently outperforms FLUX VAE under the same training steps. Overall, our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.

In the real world, fashion is about layering: adding a jacket over a shirt, or a sequence of adding and removing layers, rather than just a single-layer swap. This fundamental real-world task remains a challenge in existing Virtual Try-On (VTON) methods, which excel at single-layer replacement but are not designed to layer or de-layer an existing outfit. This paper proposes Layering Virtual Try-On (LVTON), a layering benchmark and method that preserves an existing outfit while enabling sequential layering. We find that current VTON paradigms are fundamentally ill-equipped for LVTON, as their reliance on cloth-agnostic representations and single-item datasets discards essential layering context. Our key insight is that the LVTON challenge must be disentangled into two distinct competencies: (1) General VTON Priors (e.g., deformation, identity preservation) and (2) Specific Layering Knowledge (e.g., layering order and occlusion reasoning). First, our model obtains general VTON priors by being trained on data produced by an automatic data generation pipeline that synthesizes samples from fashion videos via segmentation and inpainting. Second, the model is fine-tuned on a small, dedicated LVTON dataset to learn the layering logic. Our method achieves state-of-the-art results on our LVTON benchmark and demonstrates superior generalizability on traditional VTON benchmarks, setting new state-of-the-art results when fine-tuned and exhibiting zero-shot capabilities.

Self-captured full-body videos are popular, but most deployments require mounted cameras, carefully-framed shots, and repeated practice. We propose a more convenient solution that enables full-body video capture using handheld mobile devices. Our approach takes as input two static photos (front and back) of you in a mirror, along with an IMU motion reference that you perform while holding your mobile phone, and synthesizes a realistic video of you performing a similar target motion. We enable rendering into a new scene, with consistent illumination and shadows. We propose a novel video diffusion-based model to achieve this. Specifically, we propose a parameter-free frame generation strategy and a multi-reference attention mechanism to effectively integrate appearance information from both the front and back selfies into the video diffusion model. Further, we introduce an image-based fine-tuning strategy to enhance frame sharpness and improve shadows and reflections generation for more realistic human-scene composition.

Portrait editing is challenging for existing techniques due to difficulties in preserving subject features like identity. In this paper, we propose a training-based method leveraging auto-generated paired data to learn desired editing while ensuring the preservation of unchanged subject features. Specifically, we design a data generation process to create reasonably good training pairs for desired editing at low cost. Based on these pairs, we introduce a Multi-Conditioned Diffusion Model to effectively learn the editing direction and preserve subject features. During inference, our model produces accurate editing mask that can guide the inference process to further preserve detailed subject features. Experiments on costume editing and cartoon expression editing show that our method achieves state-of-the-art quality, quantitatively and qualitatively.

We present a method to generate full-body selfies from photographs originally taken at arms length. Because self-captured photos are typically taken close up, they have limited field of view and exaggerated perspective that distorts facial shapes. We instead seek to generate the photo some one else would take of you from a few feet away. Our approach takes as input four selfies of your face and body, a background image, and generates a full-body selfie in a desired target pose. We introduce a novel diffusion-based approach to combine all of this information into high-quality, well-composed photos of you with the desired pose and background.

Given an input painting, we reconstruct a time-lapse video of how it may have been painted. We formulate this as an autoregressive image generation problem, in which an initially blank "canvas" is iteratively updated. The model learns from real artists by training on many painting videos. Our approach incorporates text and region understanding to define a set of painting "instructions" and updates the canvas with a novel diffusion-based renderer. The method extrapolates beyond the limited, acrylic style paintings on which it has been trained, showing plausible results for a wide range of artistic styles and genres.

Image inpainting has been researched for many years. From traditional methods to current CNN models, they all pursue two targets (structural stability and texture consistency). In this paper, we propose the multi-shift CSWin Transformer (MSCSWin Transformer) and the HSV loss to focus on colors to inpaint images for these two targets. At last, we compare our model with state-of-the-art methods on the Places2 dataset to confirm our proposed module is indeed working.

Image inpainting has been researched for years. From deeper and larger models to models that focus on global information, all of them aim to obtain results closer to reality. In this paper, we combine the stripe window and line-by-line feature shift to modify the Vision Transformer (ViT) to reduce the computation cost and obtain global information from the oblique attention. In addition, we design a new loss function to enhance the texture and colors for inpainting. At last, to validate the efficacy of our proposed model, we conduct extensive experiments on commonly seen datasets (Places2 and CelebA) compared with other state-of-the-art methods. The source code and pretrained models are available at https: //github.com/bobo0303/MSCS-Net.

Image inpainting is an important task in computer vision. As admirable methods are presented, the inpainted image is getting closer to reality. However, the result is still not good enough in the reconstructed texture and structure based on human vision. Although recent advances in computer hardware have enabled the development of larger and more complex models, there is still a need for lightweight models that can be used by individuals and small-sized institutions. Therefore, we propose a lightweight model that combines a specialized transformer with a traditional convolutional neural network (CNN). Furthermore, we have noticed most researchers only consider three primary colors (RGB) in inpainted images, but we think this is not enough. So we propose a new loss function to intensify color details. Extensive experiments on commonly seen datasets (Places2 and CelebA) validate the efficacy of our proposed model compared with other state-of-the-art methods. Index Terms: HSV color space, image inpainting, joint attention, stripe window, transformer

We describe a novel approach to decompose a single panorama of an empty indoor environment into four appearance components: specular, direct sunlight, diffuse and diffuse ambient without direct sunlight. Our system is weakly supervised by automatically generated semantic maps (with floor, wall, ceiling, lamp, window and door labels) that have shown success on perspective views and are trained for panoramas using transfer learning without any further annotations. A GAN-based approach supervised by coarse information obtained from the semantic map extracts specular reflection and direct sunlight regions on the floor and walls. These lighting effects are removed via a similar GAN-based approach and a semantic-aware inpainting step. The appearance decomposition enables multiple applications including sun direction estimation, virtual furniture insertion, floor material replacement, and sun direction change, providing an effective tool for virtual home staging. We demonstrate the effectiveness of our approach on a large and recently released dataset of panoramas of empty homes.

Near-Periodic Patterns (NPP) are ubiquitous in man-made scenes and are composed of tiled motifs with appearance differences caused by lighting, defects, or design elements. A good NPP representation is useful for many applications including image completion, segmentation, and geometric remapping. But representing NPP is challenging because it needs to maintain global consistency (tiled motifs layout) while preserving local variations (appearance differences). Methods trained on general scenes using a large dataset or single-image optimization struggle to satisfy these constraints, while methods that explicitly model periodicity are not robust to periodicity detection errors. To address these challenges, we learn a neural implicit representation using a coordinate-based MLP with single image optimization. We design an input feature warping module and a periodicity-guided patch loss to handle both global consistency and local variations. To further improve the robustness, we introduce a periodicity proposal module to search and use multiple candidate periodicities in our pipeline. We demonstrate the effectiveness of our method on more than 500 images of building facades, friezes, wallpapers, ground, and Mondrian patterns in single and multi-planar scenes.

Negative sampling plays an important role in ranking-based recommender models. However, most existing sampling methods cannot generate informative item pairs with positive and negative instances due to two limitations: 1) they merely treat observed items as positive instances, ignoring the existence of potential positive items (i.e., nonobserved items users may prefer) and the probability of observed but noisy items and 2) they fail to capture the relationship between positive and negative items during negative sampling, which may cause the unexpected selection of potential positive items. In this article, we introduce a dynamic sampling strategy to search informative item pairs. Specifically, we first sample a positive instance from all the items by leveraging the overall features of user's observed items. Then, we strategically select a negative instance by considering its correlation with the sampled positive one. Formally, we propose an item pair generative adversarial network named IPGAN, where our sampling strategy is realized in two generative models for positive and negative instances, respectively. In addition, IPGAN can also ensure that the sampled item pairs are informative relative to the ground truth by a discriminative model. What is more, we propose a batch-training approach to further enhance both user and item modeling by alleviating the special bias (noise) from different users. This approach can also significantly accelerate the process of model training compared with classical GAN method for recommendation. Experimental results on three real data sets show that our approach outperforms other state-of-the-art approaches in terms of recommendation accuracy.

Generative adversarial networks (GANs) have achieved a big success in collaborative filtering (CF). However, existing GAN-based methods in CF still suffer from the high-sparsity and cold-start problems; in addition, they also undergo the issues of excessive space complexity or inadequate training. In this article, we propose path2rec a novel adversarial path-based recommendation model to address these limitations of existing GAN-based methods in recommendation task by naturally incorporating auxiliary information (e.g., social networks and item attributes). It is composed of two modules, 1) pathGAN and 2) path2vec. In pathGAN, we consider both explicit and implicit friends, as well as item attributes by regarding them as the source of graph construction. Then, we propose a smart walk strategy to automatically generate an optimizing path, which can effectively learn the semantic distribution of users and items. In path2vec, to fully exploit context features of the generated path, we use the Continuous Bag of Words (CBOW) model to fine-tune nodes representations learned by pathGAN. Through extensive experiments on real-world datasets, we demonstrate the effectiveness of the proposed path2rec by applying it into top-n item recommendation, which reaches better performance than other counterparts.

In recent recommendation systems, the image information of items is often used in conjunction with deep convolution network to directly learn the visual features of items. However, the existing approaches usually use only one image to represent an item. These approaches are inadequate for an item with multi-view related images. For a restaurant, it has visual information of food, drink, environment, and so on. Each view of an item can be represented by multiple images. In this paper, we propose a new factorization model that combines multi-view visual information with the implicit feedback data for restaurant prediction and ranking. The visual features (visual information) of images are extracted by using a deep convolution network and are integrated into a collaborative filtering framework. In order to conduct personalized recommendation better, the multi-view visual features are fused through user related weights. User related weights reflect the personalized visual preference for restaurants and the weights are different and independent between users. We applied this model to make personalized recommendations for users on two real-world restaurant review datasets. Experimental results show that our model with multi-view visual information achieves better performance than models without or with only single-view visual information.

Paper recommendation is a research topic to provide users with personalized papers of interest. However, most existing approaches equally treat title and abstract as the input to learn the representation of a paper, ignoring their semantic relationship. In this paper, we regard the abstract as a sequence of sentences, and propose a two-level attentive neural network to capture: (1) the ability of each word within a sentence to reflect if it is semantically close to the words within the title. (2) the extent of each sentence in the abstract relative to the title, which is often a good summarization of the abstract document. Specifically, we propose a Long-Short Term Memory (LSTM) network with attention to learn the representation of sentences, and integrate a Gated Recurrent Unit (GRU) network with a memory network to learn the long-term sequential sentence patterns of interacted papers for both user and item (paper) modeling. We conduct extensive experiments on two real datasets, and show that our approach outperforms other state-of-the-art approaches in terms of accuracy.

Recently, GAN-based collaborative filtering methods have gained increasing attention in recommendation tasks which can learn remarkable user and item representation. However, these existing GAN-based methods mainly suffer from two limitations: (1) Their trainings are not comprehensive given the fact that the discriminator may be trained misleadingly and over-early converging since the generator may accidentally sample real items as fake ones, resulting in the emergence of contradicting labels for the same items. (2) They fail to consider implicit friends (users with the same interests.), leading to severe limitations of recommendation performance. In this paper, we propose BiGAN, an innovative bidirectional adversarial recommendation model which can alleviate the limitations mentioned above in recommendation tasks. It consists of two GANs, namely ForwardGAN and BackwardGAN. Specifically, ForwardGAN learns to generate a group of possible interacted items given a specific user, it aims to ensure that the discriminator D-f can be trained effectively. Furthermore, BackwardGAN fully exploits implicit friends with similar behaviors, then propagates them back to ForwardGAN, where a similarity exploration strategy is implemented to gain more outstanding user representation. Therefore, two GANs are trained jointly in a circle, where the augment of one GAN will enhance another one, leading to the promising user and item representation. In the experimental part, we demonstrate that our model is superior to other state-of-the-art recommenders.

In this paper, we propose a new factorization model that combines multi-view visual feature information with the implicit feedback data for prediction and ranking. The visual information is integrated into a collaborative filtering framework. The visual features of images are extracted by using a deep neural network. In order to conduct personalized recommendation better, the multi-view visual features are fused through user related weights. The user related weights reflect the personalized visual preference for items. They are different and independent between users. Experimental results show that our model with multi-view visual information achieves the better performance than models without or with only single-view visual information.