Knowledge-intensive programming Q&A is an active research area in industry. Its application boosts developer productivity by aiding developers in quickly finding programming answers from the vast amount of information on the Internet. In this study, we propose ProQANS and its variants ReProQANS and ReAugProQANS to tackle programming Q&A. ProQANS is a neural search approach that leverages unlabeled data on the Internet (such as StackOverflow) to mitigate the cold-start problem. ReProQANS extends ProQANS by utilizing reformulated queries with a novel triplet loss. We further use an auxiliary generative model to augment the training queries, and design a novel dual triplet loss function to adapt these generated queries, to build another variant of ReProQANS termed as ReAugProQANS. In our empirical experiments, we show ReProQANS has the best performance when evaluated on the in-domain test set, while ReAugProQANS has the superior performance on the out-of-domain real programming questions, by outperforming the state-of-the-art model by up to 477% lift on the MRR metric respectively. The results suggest their robustness to previously unseen questions and its wide application to real programming questions.

In this paper we present a Click-through-rate (CTR) prediction model for product advertisement at Amazon. CTR prediction is challenging because the model needs to a) learn from text and numeric features, b) maintain low-latency at inference time, and c) adapt to a temporal advertisement distribution shift. Our proposed model is DCAF-BERT, a novel lightweight cache-friendly factorized model that consists of twin-structured BERT-like encoders for text with a mechanism for late fusion for tabular and numeric features. The factorization of the model allows for compartmentalised retraining which enables the model to easily adapt to distribution shifts. The twin encoders are carefully trained to leverage historical CTR data, using a large pre-trained language model and cross-architecture knowledge distillation (KD). We empirically find the right combination of pretraining, distillation and fine-tuning strategies for teacher and student which leads to a 1.7% ROC-AUC lift over the previous best model offline. In an online experiment we show that our compartmentalised refresh strategy boosts the CTR of DCAF-BERT by 3.6% on average over the baseline model consistently across a month.

While pre-trained large language models (LLM) like BERT have achieved state-ofthe-art in several NLP tasks, their performance on tasks with additional grounding e.g. with numeric and categorical features is less studied. In this paper, we study the application of pre-trained LLM for Click-through-rate (CTR) prediction for product advertisement in e-commerce. This is challenging because the model needs to a) learn from language as well as tabular data features, b) maintain low-latency (<5 ms) at inference time, and c) adapt to constantly changing advertisement distribution. We first show that scaling the pre-trained language model to 1.5 billion parameters significantly improves performance over conventional CTR baselines. We then present CTR-BERT, a novel lightweight cache-friendly factorized model for CTR prediction that consists of twin-structured BERT-like encoders for text with a mechanism for late fusion for text and tabular features. We train the CTR-BERT model using cross-architecture knowledge distillation (KD) and empirically study the interaction between KD and distribution shift in this setting, by a) experimenting with pre-training, distillation pre-finetuning and fine-tuning strategies b) factorizing features based on their distribution shift time scales, that allows the model to readily adapt and be re-trained. Finally, we show that CTR-BERT significantly outperforms a traditional CTR baseline with a 2.3% relative ROC-AUC lift in offline experiments and a 2% CTR lift in an online experiment.

Pre-training and then fine-tuning large language models is commonly used to achieve state-of-the-art performance in natural language processing (NLP) tasks. However, most pre-trained models suffer from low inference speed. Deploying such large models to applications with latency constraints is challenging. In this work, we focus on accelerating the inference via conditional computations. To achieve this, we propose a novel idea, Magic Pyramid (MP), to reduce both width-wise and depth-wise computation via token pruning and early exiting for Transformer-based models, particularly BERT. The former manages to save the computation via removing non-salient tokens, while the latter can fulfill the computation reduction by terminating the inference early before reaching the final layer, if the exiting condition is met. Our empirical studies demonstrate that compared to previous state of arts, MP is not only able to achieve a speed-adjustable inference but also to surpass token pruning and early exiting by reducing up to 70% giga floating point operations (GFLOPs) with less than 0.5% accuracy drop. Token pruning and early exiting express distinctive preferences to sequences with different lengths. However, MP is capable of achieving an average of 8.06x speedup on two popular text classification tasks, regardless of the sizes of the inputs.

Fast k-Nearest Neighbor search over real-valued vector spaces (KNN) is an important algorithmic task for information retrieval and recommendation systems. We present a method for using reduced precision to represent vectors through quantized integer values, enabling both a reduction in the memory overhead of indexing these vectors and faster distance computations at query time. While most traditional quantization techniques focus on minimizing the reconstruction error between a point and its uncompressed counterpart, we focus instead on preserving the behavior of the underlying distance metric. Furthermore, our quantization approach is applied at the implementation level and can be combined with existing KNN algorithms. Our experiments on both open source and proprietary datasets across multiple popular KNN frameworks validate that quantized distance metrics can reduce memory by 60% and improve query throughput by 30%, while incurring only a 2% reduction in recall.

Software metrics rarely follow a normal distribution. Therefore, software metrics are usually transformed prior to building a defect prediction model. To the best of our knowledge, the impact that the transformation has on cross-project defect prediction models has not been thoroughly explored. A cross-project model is built from one project and applied on another project. In this study, we investigate if cross-project defect prediction is affected by applying different transformations (i.e., log and rank transformations, as well as the Box-Cox transformation). The Box-Cox transformation subsumes log and other power transformations (e.g., square root), but has not been studied in the defect prediction literature. We propose an approach, namely Multiple Transformations (MT), to utilize multiple transformations for cross-project defect prediction. We further propose an enhanced approach MT+ to use the parameter of the Box-Cox transformation to determine the most appropriate training project for each target project. Our experiments are conducted upon three publicly available data sets (i.e., AEEEM, ReLink, and PROMISE). Comparing to the random forest model built solely using the log transformation, our MT+ approach improves the F-measure by 7, 59 and 43% for the three data sets, respectively. As a summary, our major contributions are three-fold: 1) conduct an empirical study on the impact that data transformation has on cross-project defect prediction models; 2) propose an approach to utilize the various information retained by applying different transformation methods; and 3) propose an unsupervised approach to select the most appropriate training project for each target project.

Source code examples are used by developers to implement unfamiliar tasks by learning from existing solutions. To better support developers in finding existing solutions, code search engines are designed to locate and rank code examples relevant to user’s queries. Essentially, a code search engine provides a ranking schema, which combines a set of ranking features to calculate the relevance between a query and candidate code examples. Consequently, the ranking schema places relevant code examples at the top of the result list. However, it is difficult to determine the configurations of the ranking schemas subjectively. In this paper, we propose a code example search approach that applies a machine learning technique to automatically train a ranking schema. We use the trained ranking schema to rank candidate code examples for new queries at run-time. We evaluate the ranking performance of our approach using a corpus of over 360,000 code snippets crawled from 586 open-source Android projects. The performance evaluation study shows that the learning-to-rank approach can effectively rank code examples, and outperform the existing ranking schemas by about 35.65 % and 48.42 % in terms of normalized discounted cumulative gain (NDCG) and expected reciprocal rank (ERR) measures respectively.

The number of mobile applications has increased drastically in the past few years. A recent study has shown that reusing source code is a common practice for Android application development. However, reuse in mobile applications is not necessarily limited to the source code (i.e., program logic). User interface (UI) design plays a vital role in constructing the user-perceived quality of a mobile application. The user-perceived quality reflects the users' opinions of a product. For mobile applications, it can be quantified by the number of downloads and raters. In this study, we extract commonly used UI elements, denoted as Common Element Sets (CESs), from user interfaces of applications. Moreover, we highlight the characteristics of CESs that can result in a high user-perceived quality by proposing various metrics. Through an empirical study on 1292 mobile applications, we observe that (i) CESs of mobile applications widely occur among and across different categories; (ii) certain characteristics of CESs can provide a high user-perceived quality; and (iii) through a manual analysis, we recommend UI templates that are extracted and summarized from CESs for developers. Developers and quality assurance personnel can use our guidelines to improve the quality of mobile applications. (C) 2016 Elsevier Inc. All rights reserved.

When encountering unfamiliar programming tasks (e.g., connecting to a database), there is a need to seek potential working code examples. Instead of using code search engines, software developers usually post related programming questions on online Q&A forums (e.g., Stack Overflow). One possible reason is that existing code search engines would return effective code examples only if a query contains identifiers (e.g., class or method names). In other words, existing code search engines do not handle natural-language queries well (e.g., a description of a programming task). However, developers may not know the appropriate identifiers at the time of the search. As the demand of searching code examples is increasing, it is of significant interest to enhance code search engines. We conjecture that expanding natural-language queries with their semantically related identifiers has a great potential to enhance code search engines. In this paper, we propose an automated approach to find identifiers (in particular API class-names) that are semantically related to a given natural-language query. We evaluate the effectiveness of our approach using 74 queries on a corpus of 23,677,216 code snippets that are extracted from 24,666 open source Java projects. The results show that our approach can effectively recommend semantically related API class-names to expand the original natural-language queries. For instance, our approach successfully retrieves relevant code examples in the top 10 retrieved results for 76 percent of 74 queries, while it is 36 percent when using the original natural-language query; and the median rank of the first relevant code example is increased from 22 to 7.

The number of mobile applications (apps) and mobile devices has increased considerably over the past few years. Online app markets, such as the Google Play Store, use a star-rating mechanism to quantify the user-perceived quality of mobile apps. Users may rate apps on a five point (star) scale where a five star-rating is the highest rating. Having considered the importance of a high star-rating to the success of an app, recent studies continue to explore the relationship between the app attributes, such as User Interface (UI) complexity, and the user-perceived quality. However, the user-perceived quality reflects the users' experience using an app on a particular mobile device. Hence, the user-perceived quality of an app is not solely determined by app attributes. In this paper, we study the relation of both device attributes and app attributes with the user-perceived quality of Android apps from the Google Play Store. We study 20 device attributes, such as the CPU and the display size, and 13 app attributes, such as code size and UI complexity. Our study is based on data from 30 types of Android mobile devices and 280 Android apps. We use linear mixed effect models to identify the device attributes and app attributes with the strongest relationship with the user-perceived quality. We find that the code size has the strongest relationship with the user-perceived quality. However, some device attributes, such as the CPU, have stronger relationships with the user-perceived quality than some app attributes, such as the number of UI inputs and outputs of an app. Our work helps both device manufacturers and app developers. Manufacturers can focus on the attributes that have significant relationships with the user-perceived quality. Moreover, app developers should be careful about the devices for which they make their apps available because the device attributes have a strong relationship with the ratings that users give to apps.

Software defects can lead to undesired results. Correcting defects costs 50 % to 75 % of the total software development budgets. To predict defective files, a prediction model must be built with predictors (e.g., software metrics) obtained from either a project itself (within-project) or from other projects (cross-project). A universal defect prediction model that is built from a large set of diverse projects would relieve the need to build and tailor prediction models for an individual project. A formidable obstacle to build a universal model is the variations in the distribution of predictors among projects of diverse contexts (e.g., size and programming language). Hence, we propose to cluster projects based on the similarity of the distribution of predictors, and derive the rank transformations using quantiles of predictors for a cluster. We fit the universal model on the transformed data of 1,385 open source projects hosted on SourceForge and GoogleCode. The universal model obtains prediction performance comparable to the within-project models, yields similar results when applied on five external projects (one Apache and four Eclipse projects), and performs similarly among projects with different context factors. At last, we investigate what predictors should be included in the universal model. We expect that this work could form a basis for future work on building a universal model and would lead to software support tools that incorporate it into a regular development workflow.

Ontological context model that can be instantiated for the software domain.Context that supports information sharing and reuse among tools within the software domain.Application examples for the use of the context model in the software domain.Evaluation of the model to illustrate the applicability for mentor assignment. Recent years have witnessed rapid advances in the use of contextual information in ubiquitous and ambient computing. Such information improves situated cognition and awareness as well as stakeholders' usage experience. While domains such as Web 3.0 - the next generation of the web - have made context-awareness a main requirement of their solution space, the software engineering domain still lacks the same rate of adoption. In our research, we introduce an ontology based context-aware meta-model that takes advantage of Semantic Web technologies to capture and formalize context information. Providing such formal context representation allows us to make context information an integrated and reusable part of the software engineering domain. We present several case studies related to the software evolution domain to illustrate the benefit of sharing and reusing context for various software engineering tasks, such as mentor recommendation, code search, and result ranking.

Application Programming Interfaces (APIs) facilitate pragmatic reuse and improve the productivity of software development. An API usage pattern documents a set of method calls from multiple API classes to achieve a reusable functionality. Existing approaches often use frequent-sequence mining to extract API usage patterns. However, as reported by earlier studies, frequent-sequence mining may not produce a complete set of usage patterns. In this paper, we explore the possibility of mining API usage patterns without relying on frequent-pattern mining. Our approach represents the source code as a network of object usages where an object usage is a set of method calls invoked on a single API class. We automatically extract usage patterns by clustering the data based on the co-existence relations between object usages. We conduct an empirical study using a corpus of 11,510 Android applications. The results demonstrate that our approach can effectively mine API usage patterns with high completeness and low redundancy. We observe 18% and 38% improvement on F-measure and response time respectively comparing to usage pattern extraction using frequent-sequence mining. (C) 2016 Elsevier Inc. All rights reserved.

Code clones are unavoidable entities in software ecosystems. A variety of clone-detection algorithms are available for finding code clones. For Type-3 clone detection at method granularity (i.e., similar methods with changes in statements), dissimilarity threshold is one of the possible configuration parameters. Existing approaches use a single threshold to detect Type-3 clones across a repository. However, our study shows that to detect Type-3 clones at method granularity on a large-scale heterogeneous repository, multiple thresholds are often required. We find that the performance of clone detection improves if selecting different thresholds for various groups of clones in a heterogeneous repository (i.e., various applications). In this paper, we propose a threshold-free approach to detect Type-3 clones at method granularity across a large number of applications. Our approach uses an unsupervised learning algorithm, i.e., k-means, to determine true and false clones. We use a clone benchmark with 330,840 tagged clones from 24,824 open source Java projects for our study. We observe that our approach improves the performance significantly by 12% in terms of F-measure. Furthermore, our threshold-free approach eliminates the concern of practitioners about possible misconfiguration of Type-3 clone detection tools.

Big data analysis is an emerging research topic in various domains, and clone detection is no exception. The goal is to create big data inter‐project clone corpora across open‐source or corporate‐source code repositories. Such corpora can be used to study developer behavior and to reduce engineering costs by extracting globally duplicated efforts into new APIs and as a basis for code completion and API usage support. However, building scalable clone detection tools is challenging. It is often impractical to use existing state‐of‐the‐art tools to analyze big data because the memory and execution time required exceed the average user's resources. Some tools have inherent limitations in their data structures and algorithms that prevent the analysis of big data even when extraordinary resources are available. These limitations are impossible to overcome if the source code of the tool is unavailable or if the user lacks the time or expertise to modify the tool without harming its performance or accuracy. In this research, we have investigated the use of our shuffling framework for scaling classical clone detection tools to big data. The framework achieves scalability on commodity hardware by partitioning the input dataset into subsets manageable by the tool and computing resources. A non‐deterministic process is used to randomly ‘shuffle’ the contents of the dataset into a series of subsets. The tool is executed for each subset, and its output for each is merged into a single report. This approach does not require modification to the subject tools, allowing their individual strengths and precision to be captured at an acceptable loss of recall. In our study, we explored the performance and applicability of the framework for the big data dataset, IJaDataset 2.0, which consists of 356 million lines of code from 25,000 open‐source Java projects. We begin with a computationally inexpensive version of our framework based on pure random shuffling. This version was successful at scaling the tools to IJaDataset but required many subsets to achieve a desirable recall. Using our findings, we incrementally improved the framework to achieve a satisfactory recall using fewer resources. We investigated the use of efficient file tracking and file‐similarity heuristics to bias the shuffling algorithm toward subsets of the dataset that contain undetected clone pairs. These changes were successful in improving the recall performance of the framework. Our study shows that the framework is able to achieve up to 90–95% of a tool's native recall using standard hardware. Copyright © 2014 John Wiley & Sons, Ltd.

Web service composition enables seamless and dynamic integration of Web services. The behavior of participant Web services determines the overall performance of a composition. Therefore, it is important to choose high quality services for service composition. Existing Web service selection and discovery approaches rely on non-functional aspects (also known as quality of service or QoS), e.g., response time and availability. Though these parameters are crucial for selecting Web services, they may not reflect the user's perspective of quality. In this paper, we explore the feasibility of incorporating perceived quality from user's perspective for service selection and composition. We name such quality attributes as quality of experience (QoE). First, we propose a solution that automatically mines and identifies QoE attributes from the Web. Second, we study the application of such dynamically extracted QoE attributes for service selection. For the evaluation purpose, we collected more than 34,000 reviews from 58 different services in six domains. Our findings show that it is possible to automatically identify QoE attributes with an average precision and recall of 92 and 80 percent respectively. Our study shows that there is a strong positive correlation between QoS and QoE. Hence QoE can be used during service selection especially when QoS data are not available. Furthermore, we found 70 percent of service discovery queries indeed contain QoE attributes showing the importance of QoE attributes during the service discovery phase.

Recently, automatic code comment generation is proposed to facilitate program comprehension. Existing code comment generation techniques focus on describing the functionality of the source code. However, there are other aspects such as insights about quality or issues of the code, which are overlooked by earlier approaches. In this paper, we describe a mining approach that recommends insightful comments about the quality, deficiencies or scopes for further improvement of the source code. First, we conduct an exploratory study that motivates crowdsourced knowledge from Stack Overflow discussions as a potential resource for source code comment recommendation. Second, based on the findings from the exploratory study, we propose a heuristic-based technique for mining insightful comments from Stack Overflow Q & A site for source code comment recommendation. Experiments with 292 Stack Overflow code segments and 5,039 discussion comments show that our approach has a promising recall of 85.42%. We also conducted a complementary user study which confirms the accuracy and usefulness of the recommended comments.

End-users conduct various on-line activities. Quite often they re-visit websites and use services to perform repeated activities, such as on-line shopping. The end-users are required to enter the same information into various web services to accomplish such repeated tasks. Typing redundant information repetitively into such services negatively impacts the user experience. In this study, we propose an approach to prevent end-users from such unnecessary interruption. Our approach propagates user inputs across services by linking similar input and output parameters. Our approach also pre-fills values to the input parameters which could not be filled by the values from other input or output parameters. We propose a meta-data model for storing user inputs and an Input Parameter Context Model for identifying similar input or output parameters. We have implemented our approach and evaluated it on the real world services through an empirical study. Our overall approach can reduce on average 37% of input parameters through the execution of composed services.