User feedback on software usage is utilised by developers to improve their software. Software product forums are platforms rich in software-related user feedback, such as forum threads containing bug reports or requests for new features. However, previous studies have mainly focused on analysing user feedback from software product forums as individual sentences, which can lead to missing insights and a lack of understanding of the overall context of forum posts. To fill this gap in research, this work examines user feedback found in software product forum posts to investigate the differences between content classifications found in forum sentences and posts. We manually evaluated software product forum posts collected from two open-sourced software product forums and discovered five new types of user feedback that can only be identified when examining user feedback in the form of forum posts. Additionally, we examined the association between sentence classifications found within software product forums. Our results indicate that contextual information complimenting product improvement insights can be found in software product forums, with a confidence of 0.75 and 0.69 for the association between apparent bug and application usage sentences. This information can be used to reduce manual efforts required to chase up missing contextual information when attempting to understand or fix software issues. We also provide insights into the progression of posts in software product forums at the thread-level, and our progression flowchart can be used to summarise the sequence of events in software product forum threads. Our findings reveal the importance of looking at user feedback within software product forums in the format of forum posts to identify new insights on user feedback for software improvements.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Training Large Language Models (LLMs) with Reinforcement Learning from AI Feedback (RLAIF) aligns model outputs more closely with human preferences. This involves an evaluator model ranking multiple candidate responses to user prompts. However, the rankings from popular evaluator models such as GPT-4 can be inconsistent. We propose the Repeat Ranking method - where we evaluate the same responses multiple times and train only on those responses which are consistently ranked. Using 2,714 prompts in 62 languages, we generated responses from 7 top multilingual LLMs and had GPT-4 rank them five times each. Evaluating on MT-Bench chat benchmarks in six languages, our method outperformed the standard practice of training on all available prompts. Our work highlights the quality versus quantity trade-off in RLAIF dataset generation and offers a stackable strategy for enhancing dataset and thus model quality.

Open source large language models (LLMs) have shown great improvements in recent times. However, many of these models are focused solely on popular spoken languages. We present a high quality dataset of more than 70k prompt-response pairs in 74 languages which consist of human generated prompts and synthetic responses. We use this dataset to train a state-of-the-art open source English LLM to chat multilingually. We evaluate our model on MT-Bench chat benchmarks in 6 languages, finding that our multilingual model outperforms previous state-of-the-art open source LLMs across each language. We further find that training on more multilingual data is beneficial to the performance in a chosen target language (Japanese) compared to simply training on only data in that language. These results indicate the necessity of training on large amounts of high quality multilingual data to make a more accessible LLM.

Understanding users’ needs is crucial to building and maintaining high quality software. Online software user feedback has been shown to contain large amounts of information useful to requirements engineering (RE). Previous studies have created machine learning classifiers for parsing this feedback for development insight. While these classifiers report generally good performance when evaluated on a test set, questions remain as to how well they extend to unseen data in various forms. This study evaluates machine learning classifiers’ performance on feedback for two common classification tasks (classifying bug reports and feature requests). Using seven datasets from prior research studies, we investigate the performance of classifiers when evaluated on feedback from different apps than those contained in the training set and when evaluated on completely different datasets (coming from different feedback channels and/or labelled by different researchers). We also measure the difference in performance of using channel-specific metadata as a feature in classification. We find that using metadata as features in classifying bug reports and feature requests does not lead to a statistically significant improvement in the majority of datasets tested. We also demonstrate that classification performance is similar on feedback from unseen apps compared to seen apps in the majority of cases tested. However, the classifiers evaluated do not perform well on unseen datasets. We show that multi-dataset training or zero shot classification approaches can somewhat mitigate this performance decrease. We discuss the implications of these results on developing user feedback classification models to analyse and extract software requirements.

Software products now have more users than ever. This means more people to please, more use-cases to consider, and more requirements to fulfill. These users can then write feedback on software in any number of public or private online repositories. Many tools have been proposed for classifying, embedding, clustering, and characterizing this feedback in aid of generating requirements from it. I am investigating which techniques and machine learning models are most appropriate for enabling these analyses across multiple feedback platforms and data domains.

User feedback is an important resource in modern software development, often containing requirements that help address user concerns and desires for a software product. The feedback in online channels is a recent focus for software engineering researchers, with multiple studies proposing automatic analysis tools. In this work, we investigate the product forums of two large open source software projects. Through a quantitative analysis, we show that forum feedback is often manually linked to related issue tracker entries and product documentation. By linking feedback to their existing documentation, development teams enhance their understanding of known issues, and direct their users to known solutions. We discuss how the links between forum, issue tracker, and product documentation form a requirements ecosystem that has not been identified in the previous literature. We apply state-of-the-art deep-learning to automatically match forum posts with related issue tracker entries. Our approach identifies requirement matches with a mean average precision of 58.9% and hit ratio of 82.2%. Additionally, we apply deep-learning using an innovative clustering technique, achieving promising performance when matching forum posts to related product documentation. We discuss the possible applications of these automated techniques to support the flow of requirements between forum, issue tracker, and product documentation.

Knowing the topics of a software forum post, such as those on StackOverflow, allows for greater analysis and understanding of the large amounts of data that come from these communities. One approach to this problem is using extreme multi label classification (XMLC) to predict the topic (or “tag”) of a post from a potentially very large candidate label set. While previous work has trained these models on data which has explicit text-to-tag information, we assess the classification ability of embedding models which have not been trained using such structured data (and are thus “unsupervised”) to assess the potential applicability to other forums or domains in which tag data is not available.We evaluate 14 unsupervised pre-trained models on 0.1% of all StackOverflow posts against all 61,662 possible StackOverflow tags. We find that an MPNet model trained partially on unlabelled StackExchange data (i.e. without tag data) achieves the highest score overall for this task, with a recall score of 0.161 R@1. These results inform which models are most appropriate for use in XMLC of StackOverflow posts when supervised training is not feasible. This offers insight into these models’ applicability in similar but not identical domains, such as software product forums. These results suggest that training embedding models using in-domain title-body or question-answer pairs can create an effective zero-shot topic classifier for situations where no topic data is available.

The software industry lacks gender diversity. Recent research has suggested that a toxic working culture is to blame. Studies have found that communications in software repositories directed towards women are more negative in general. In this study, we use a destructive criticism lens to examine gender differences in software code review feedback. Software code review is a practice where code is peer reviewed and negative feedback is often delivered. We explore differences in perceptions, frequency, and impact of destructive criticism across genders. We surveyed 93 software practitioners eliciting perceived reactions to hypothetical scenarios (or vignettes) where participants are asked to imagine receiving either constructive or destructive criticism. In addition, the survey collected general opinions on feedback obtained during software code review as well as the frequency that participants give and receive destructive criticism. We found that opinions on destructive criticism vary. Women perceive destructive criticism as less appropriate and are less motivated to continue working with the developer after receiving destructive criticism. Destructive criticism is fairly common with more than half of respondents having received nonspecific negative feedback and nearly a quarter having received inconsiderate negative feedback in the past year. Our results suggest that destructive criticism in code review could be a contributing factor to the lack of gender diversity observed in the software industry.

Feedback from software users is vital for engineering better software requirements. One tool for extracting requirements from online user feedback is clustering, where the most mentioned topics are found by grouping similar feedback together. For these topics to be understood, clusters have been summarized in previous work using characterizing phrases or sentences. This work evaluates which method of characterization (unigrams, bigrams, trigrams, or sentences) is most effective for understanding the semantic meaning of a whole cluster using feedback from multiple feedback sources. We evaluate multiple characterization methods to determine the ability of each method to create distinct, descriptive characterizations. We further evaluate the amount of requirements relevant characterizations created by each characterization method. We find that unigrams, bigrams, trigrams, and full sentences all perform similarly in distinguishing clusters from each other. However, we find that fewer and more expressive characterizations, such as full sentences, contain more requirements relevant information from a feedback cluster compared to more numerous but less expressive unigrams, meaning a sentence will better summarize the important requirement relevant information from a cluster. Our findings inform the future development of user feedback clustering tools, with different cluster characterization methods being quantitatively measured for the first time.

Software product forums is a platform filled with user feedback that utilises the sub-forum feature to categorise user discussion into themes. These sub-forums are very similar to classification labels that have been used to automatically classify user feedback on other platforms such as Troubleshooting and Feature Request. It would be very beneficial to the CrowdRE community if these sub-forum categories can be utilised in a research setting as it would reduce the effort required to label content for classification manually. However, no research has been done on the accuracy of these sub-forum categorisations in software product forums. In this exploratory study, we examined the accuracy of user categorised posts in two software product forums and discovered that users incorrectly categorise more than 20% of the posts during submission. Our discovery suggests that at the current stage, sub-forum categories should not be trusted as a label to classify feedback automatically.

User feedback on software products has been shown to be useful for development and can be exceedingly abundant online. Many approaches have been developed to elicit requirements in different ways from this large volume of feedback, including the use of unsupervised clustering, underpinned by text embeddings. Methods for embedding text can vary significantly within the literature, highlighting the lack of a consensus as to which approaches are best able to cluster user feedback into requirements relevant groups. This work proposes a methodology for comparing text embeddings of user feedback using existing labelled datasets. Using 7 diverse datasets from the literature, we apply this methodology to evaluate both established text embedding techniques from the user feedback analysis literature (including topic modelling and word embeddings) as well as text embeddings from state of the art deep text embedding models. Results demonstrate that text embeddings produced by state of the art models, most notably the Universal Sentence Encoder (USE), group feedback with similar requirements relevant characteristics together better than other evaluated techniques across all seven datasets. These results can help researchers select appropriate embedding techniques when developing future unsupervised clustering approaches within user feedback analysis.