Evaluating Design Variations Using UX-Analyzer

User Experience (UX) is an essential factor in today's business, and so large companies have adopted a process of continuous UX monitoring. In this process, design experts run user tests and analyze UX performance indicators, which are usually obtained from standardized questionnaires performed with users. Thus, this process is expensive, as it requires paying test participants, and allocating the time to manually analyze the collected data. Previously, we have defined interaction effort as a metric for the dynamic performance of web elements, based on measures of the user interaction, and we presented UX-Analyzer as a tool to visualize the interaction effort. UX-Analyzer allows UX experts or other team members to evaluate web pages automatically and in a transparent way. Visualizing the interaction effort may provide a good indication towards the level of UX of an online system. It may also be used in the context of an A/B testing approach that instead of revenue or conversions, it compares the UX of alternative versions of a website. In this paper we show how UX-Analyzer can perform in real settings. We present a case study with 152 participants that show its applicability, and we also add a qualitative study with design professionals to assess its adoption and usefulness. Additionally, we have extended the related work, and describe some new features that improve the tool's flexibility.