Neural Design Network: Graphic Layout Generation with Constraints

Cited by: 0|Bibtex|Views36
Other Links: arxiv.org
Weibo:
We present examples of constructing real designs based on generated layouts, and an application of layout recommendation

Abstract:

Graphic design is essential for visual communication with layouts being fundamental to composing attractive designs. Layout generation differs from pixel-level image synthesis and is unique in terms of the requirement of mutual relations among the desired components. We propose a method for design layout generation that can satisfy user...More

Code:

Data:

0
Introduction
  • Graphic design is essential for visual communication with layouts being fundamental to composing attractive designs.
  • The authors propose a method for design layout generation that can satisfy userspecified constraints.
  • The third module finetunes the predicted layout.
  • The authors construct real designs based on predicted layouts for a better understanding of the visual quality.
  • Log in Cancel posed method can generate layouts with good visual quality, Logo even with no constraint provided.
  • To better visualize the quality of the generated layouts, the authors present designs with real assets generated from the predicted layouts.
  • The authors can constrain the size of specific components to desired shapes using the augmented layout generation module
Highlights
  • Graphic design is essential for visual communication with layouts being fundamental to composing attractive designs
  • We introduce neural design network (NDN), a new approach of synthesizing a graphic design layout given a set of components with user-specified attributes and constraints
  • We introduce the neural design network using graph neural network and conditional variational auto-encoder (VAE) [18, 28] with the goal of capturing better representations of design layouts
  • We evaluate the visual quality following Frechet Inception Distance (FID) [9] by measuring how close the distribution of generated layout is to the real ones
  • Extensive quantitative and qualitative experiments demonstrate the efficacy of the proposed model
  • We present examples of constructing real designs based on generated layouts, and an application of layout recommendation
Methods
  • The authors perform the evaluation on the following algorithms:

    sg2im [14]. The model is proposed to generate a natural scene layout from a given scene graph.
  • The sg2im method takes as inputs graphs with complete relations in the setting where all constraints are provided.
  • When the authors compare with this method in the setting where no constraint is given, the authors simplify the input scene graph by removing all relations.
  • LayoutVAE [15]
  • This model takes a label set as input, and predicts the number of components for each label as well as the locations of each component.
  • The authors refer to LayoutVAE-loo as the model that predicts the bounding box of a single component when all other components are provided and fixed
Conclusion
  • The authors propose a neural design network to handle design layout generation given user-specified con-

    Headline Text Text straints.
  • The authors propose a neural design network to handle design layout generation given user-specified con-.
  • Headline Text Text straints.
  • The proposed method can generate layouts that are visually appealing and follow the constraints with a threemodule framework, including a relation prediction module, a layout generation module, and a refinement module.
  • Extensive quantitative and qualitative experiments demonstrate the efficacy of the proposed model.
  • The authors present examples of constructing real designs based on generated layouts, and an application of layout recommendation
Summary
  • Introduction:

    Graphic design is essential for visual communication with layouts being fundamental to composing attractive designs.
  • The authors propose a method for design layout generation that can satisfy userspecified constraints.
  • The third module finetunes the predicted layout.
  • The authors construct real designs based on predicted layouts for a better understanding of the visual quality.
  • Log in Cancel posed method can generate layouts with good visual quality, Logo even with no constraint provided.
  • To better visualize the quality of the generated layouts, the authors present designs with real assets generated from the predicted layouts.
  • The authors can constrain the size of specific components to desired shapes using the augmented layout generation module
  • Methods:

    The authors perform the evaluation on the following algorithms:

    sg2im [14]. The model is proposed to generate a natural scene layout from a given scene graph.
  • The sg2im method takes as inputs graphs with complete relations in the setting where all constraints are provided.
  • When the authors compare with this method in the setting where no constraint is given, the authors simplify the input scene graph by removing all relations.
  • LayoutVAE [15]
  • This model takes a label set as input, and predicts the number of components for each label as well as the locations of each component.
  • The authors refer to LayoutVAE-loo as the model that predicts the bounding box of a single component when all other components are provided and fixed
  • Conclusion:

    The authors propose a neural design network to handle design layout generation given user-specified con-

    Headline Text Text straints.
  • The authors propose a neural design network to handle design layout generation given user-specified con-.
  • Headline Text Text straints.
  • The proposed method can generate layouts that are visually appealing and follow the constraints with a threemodule framework, including a relation prediction module, a layout generation module, and a refinement module.
  • Extensive quantitative and qualitative experiments demonstrate the efficacy of the proposed model.
  • The authors present examples of constructing real designs based on generated layouts, and an application of layout recommendation
Tables
  • Table1: Quantitative comparisons. We compare the proposed method to other works on three datasets using three settings: no-constraint setting that no prior constraint is provided (first row), all-constraint setting that all relations are provided (second row), and leave-one-out setting that aims to predict the bounding box of a component with ground-truth bounding boxes of other components provided. The FID metric measures the realism and diversity, the alignment metric measures the alignment among components, and the accuracy metric measures the prediction accuracy in the leave-one-out setting
  • Table2: Ablation on partial constraints. We measure the FID and alignment of the proposed method taking different percentages of prior constraints as inputs using the RICO dataset. We also show that the refinement module can further improve the visual quality as well as the alignment
Download tables as Excel
Related work
  • Natural scene layout generation. Layout is often used as the intermediate representation in image generation task conditioned on text [8, 10, 31] or scene graph [14]. Instead of directly learning the mapping from the source domain (e.g., text and scene graph) to the image domain, these methods model the operation as a two-stage framework. They first predict layouts conditioned on the input sources, and then generate images based on the predicted layouts. Recently, Jyothi et al propose the LayoutVAE [15], which is a generative framework that can synthesize scene layout given a set of labels. However, a graphic design layout has several fundamental differences to a natural scene layout. The demands for relationship and alignment among components are strict in graphic design. A few pixels offsets of components can either cause a difference in visual experience or even ruin the whole design. The graphic design layout does not only need to look realistic but also needs to consider the aesthetic perspective.
Reference
  • Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. Learning visual importance for graphic designs and data visualizations. In UIST, 2017. 2
    Google ScholarFindings
  • Niranjan Damera-Venkata, Jose Bento, and Eamonn O’Brien-Strain. Probabilistic document model for automated document composition. In DocEng, 2011. 2
    Google ScholarLocate open access versionFindings
  • Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building datadriven design applications. In UIST, 2017. 2, 5
    Google ScholarLocate open access versionFindings
  • David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NIPS, 2015. 3
    Google ScholarLocate open access versionFindings
  • Christoph Goller and Andreas Kuchler. Learning taskdependent distributed representations by backpropagation through structure. In ICNN, 1996. 3
    Google ScholarLocate open access versionFindings
  • Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014. 2
    Google ScholarLocate open access versionFindings
  • Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In IJCNN, 2005. 3
    Google ScholarLocate open access versionFindings
  • Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to compositions to videos. In ECCV, 2012
    Google ScholarLocate open access versionFindings
  • Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017. 6
    Google ScholarLocate open access versionFindings
  • Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical textto-image synthesis. In CVPR, 2018. 2
    Google ScholarLocate open access versionFindings
  • Nathan Hurst, Wilmot Li, and Kim Marriott. Review of automatic document formatting. In DocEng, 2009. 2
    Google ScholarLocate open access versionFindings
  • Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In CVPR, 2016. 3
    Google ScholarLocate open access versionFindings
  • Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to-graph translation for molecular optimization. In ICLR, 2019. 3
    Google ScholarLocate open access versionFindings
  • Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In CVPR, 2018. 2, 3, 6
    Google ScholarLocate open access versionFindings
  • Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In ICCV, 2019. 2, 6
    Google ScholarLocate open access versionFindings
  • Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 2
    Google ScholarLocate open access versionFindings
  • Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6
    Google ScholarLocate open access versionFindings
  • Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 3
    Google ScholarLocate open access versionFindings
  • Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017. 3
    Google ScholarLocate open access versionFindings
  • Ranjitha Kumar, Jerry O Talton, Salman Ahmad, and Scott R Klemmer. Bricolage: example-based retargeting for web design. In SIGCHI, 2011. 2
    Google ScholarLocate open access versionFindings
  • Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. Diverse imageto-image translation via disentangled representations. In ECCV, 2018. 4
    Google ScholarFindings
  • Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing to music. In NeurIPS, 2019. 6
    Google ScholarLocate open access versionFindings
  • Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. In ICLR, 2019. 2
    Google ScholarLocate open access versionFindings
  • Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learning design semantics for mobile apps. In UIST, 2018. 2, 5
    Google ScholarLocate open access versionFindings
  • Peter ODonovan, Aseem Agarwala, and Aaron Hertzmann. Learning layouts for single-pagegraphic designs. TVCG, 2014. 2
    Google ScholarLocate open access versionFindings
  • Xufang Pang, Ying Cao, Rynson WH Lau, and Antoni B Chan. Directing user attention via visual flow on web designs. ACM TOG (Proc. SIGGRAPH), 2016. 2
    Google ScholarLocate open access versionFindings
  • Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv:1906.00446, 2019. 2
    Findings
  • Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014. 3
    Google ScholarLocate open access versionFindings
  • Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. TNN, 2008. 3
    Google ScholarLocate open access versionFindings
  • Sou Tabata, Hiroki Yoshihara, Haruka Maeda, and Kei Yokoyama. Automatic layout generation for graphical design magazines. In SIGGRAPH, 2019. 2
    Google ScholarLocate open access versionFindings
  • Fuwen Tan, Song Feng, and Vicente Ordonez. Text2scene: Generating abstract scenes from textual descriptions. In CVPR, 2019. 2
    Google ScholarLocate open access versionFindings
  • Ying Cao Xinru Zheng, Xiaotian Qiao and Rynson W.H. Lau. Content-aware generative modeling of graphic design layouts. SIGGRAPH, 2019. 2, 5
    Google ScholarLocate open access versionFindings
  • Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV, 2018. 3
    Google ScholarLocate open access versionFindings
  • Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NIPS, 2017. 4
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments