Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps
IEEE Transactions on Software Engineering(2020)
摘要
It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks:
detection
,
classification
, and
assembly
. First, logical components of a GUI are
detected
from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately
classify
GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically
assembled
. We implemented this approach for Android in a system called
ReDraw
. Our evaluation illustrates that
ReDraw
achieves an average GUI-component classification accuracy of 91 percent and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.
更多查看译文
关键词
Graphical user interfaces,Software,Task analysis,Prototypes,Metadata,Androids,Humanoid robots
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要