Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs.

Context: Developer eXperience (Dev-X) focuses on ensuring a better experience for software developers, while still achieving development goals. Indeed, several existing studies suggest that improved Dev-X may actually result in improved development productivity. But a comprehensive understanding of how to empirically assess Dev-X remains limited. Aims and Method: This paper determines the empirical coverage of Dev-X facets to date by analyzing employed empirical designs and identifying quality issues or guidelines for assessing Dev-X. To this end, it presents a mapping study of 231 articles that employed empirical methods to explore the Dev-X landscape. Results: It identifies 16 facets of Dev-X-spanning emotions, values, and perceptions-and systematically maps the empirical methods, data types, and evaluation metrics used to assess these facets. The findings reveal gaps in facet coverage, highlight underutilized empirical designs, and stress the importance of quality assessment and methodological triangulation. This research offers a facet-wise synthesis of empirical designs and provides actionable guidelines to help researchers and practitioners empirically evaluate Dev-X. Conclusion: Based on SLR findings, this paper presents a Dev-X facet-based synthesis of empirical designs to guide selection of method, data, and metric combinations in future studies. By revealing design gaps (e.g., underexplored facets, overuse of surveys, weak corroboration) and highlighting dominant practices (e.g., survey-based ratings for emotions and values), our facet-wise schema offers an evidence-based guide for context-appropriate Dev-X evaluation strategies.

Code clones are similar code fragments that often arise from copy-and-paste programming. Neural networks can classify pairs of code fragments as clone/not-clone with high accuracy. However, finding clones in industrial-scale code needs a more scalable approach than pairwise comparison. We extend existing neural network-based clone detection schemes to handle codebases that far exceed available memory, using indexing and search methods for external storage such as disks and solid-state drives. We generate a high-dimensional vector embedding for each code fragment using a transformer-based neural network. We then find similar embeddings using efficient multidimensional nearest neighbor search algorithms on external storage to find similar embeddings without pairwise comparison. We identify specific problems with industrial-scale code bases, such as large sets of almost identical code fragments that interact poorly with k-nearest neighbour search algorithms, and provide an effective solution. We demonstrate that our disk-based clone search approach achieves similar clone detection accuracy as an equivalent in-memory technique. Using a solid-state drive as external storage, our approach is around 2 x slower than the in-memory approach for a problem size that can fit within memory. We further demonstrate that our approach can scale to over a billion lines of code, providing valuable insights into the trade-offs between indexing speed, query performance, and storage efficiency for industrial-scale code clone detection.

Context and Motivation – Developer eXperience (Dev-X) is a recent research area that focuses on developers perceptions, feelings, and values with respect to software development and software quality. Research suggests that factors and practices related to Dev-X can have a substantial impact on developer productivity (Dev-P). However, despite a large and diverse body of literature on factors that can impact Dev-P in general, there is no coherent and comprehensive characterization of how Dev-X-specific insights can influence developer productivity. Aims – In the presented research, we aim to provide a coherent, comprehensive characterization of factors and practices related to Dev-X, with a particular focus on those factors and practices that potentially affect Dev-P. Method – To this end, we performed a systematic literature review and identified 218 relevant papers in this area. We characterize the papers based on the related frameworks, and concepts common to Dev-X and Dev-P as presented in the existing literature. Dev-X factors such as ’work fragmentation’ and practices such as ’collaboration with owner-developer’ are identified using a grounded-in-the-literature, content-analysis method, guided by the theory. For each Dev-X factor, we identify attributes that might be used to assess/measure the current status (of an organisation or project) regarding that factor and how that factor and its effects on productivity have been evidenced in the literature (mentioned vs. considered in questionnaires vs. substantiated with a more positivist evaluation). Results – We identify 33 Dev-X-related factors and 41 Dev-X-related practices which are organized into ten themes to summarize their influence. The results suggest that the availability of required resources, relevant expertise re the allocated tasks, and fewer interruptions are among the top positively impacting factors. Conversely, factors like code complexity, heterogeneous contexts of tasks, and non-adherence to standardization harm Dev-X and Dev-P. Top industrial practices employed to mitigate the negative influence of factors include characterization-based task assignments, mental model support, and the timely evolution of technologies. Conclusions – Overall, this research suggests that organizations can influence Dev-P through improved Dev-X, incorporating suitable practices to mediate relevant factors in their context. Important in this regard are practices like fragmenting large tasks, highlighting the utility of proposed tasks/changes to the developers, and promoting (developer) ownership of artefacts. Finally, our results point to areas where further research seems appropriate, i.e., where Dev-X factors/practices have been proposed as being impactful on Dev-P but not yet fully substantiated or explored as such (factors like ‘Nature of Activity’ and practices like choosing practices/protocols appropriately).

Source code clones pose risks ranging from intellectual property violations to unintended vulnerabilities. Effective and efficient scalable clone detection, especially for diverged clones, remains challenging. Large language models (LLMs) have recently been applied to clone detection tasks. However, the rapid emergence of LLMs raises questions about optimal model selection and potential LLM-ensemble efficacy. This paper addresses the first question by identifying 76 LLMs and filtering them down to suitable candidates for large-scale clone detection. The candidates were evaluated on two public industrial datasets, BigCloneBench, and a commercial large-scale dataset. No uniformly 'best-LLM' emerged, though CodeT5+110M, CuBERT and SPTCode were top-performers. Analysis of LLM-candidates suggested that smaller embedding sizes, smaller tokenizer vocabularies and tailored datasets are advantageous. On commercial large-scale dataset a top-performing CodeT5+110M achieved 39.71% precision: twice the precision of previously used CodeBERT. To address the second question, this paper explores ensembling of the selected LLMs: effort-effective approach to improving effectiveness. Results suggest the importance of score normalization and favoring ensembling methods like maximum or sum over averaging. Also, findings indicate that ensembling approach can be statistically significant and effective on larger datasets: the best-performing ensemble achieved even higher precision of 46.91% over individual LLM on the commercial large-scale code.

Explainable artificial intelligence (XAI) is essential for fostering trust, transparency, and accountability in machine learning systems, particularly when applied in high-stakes domains. This paper introduces a novel XAI system designed for classification tasks on tabular data, which offers a balance between performance and interpretability. The proposed method, GlassBoost, first trains an XGBoost model on a given dataset and then computes gain scores, quantifying the average improvement in the model’s loss function contributed by each feature during tree splits. Based on these scores, a subset of significant features is selected. A shallow decision tree is then trained using the top d features with the highest gain scores, where d is significantly smaller than the total number of original features. This model compression yields a transparent, IF–THEN rule-based decision process that remains faithful to the original high-performing model. To evaluate the system, we apply it to an anomaly detection task in the context of intrusion detection systems (IDSs), using a dataset containing traffic features from both malicious and normal activities. Results show that our method achieves high accuracy, precision, and recall while providing a clear and interpretable explanation of its decision-making. We further validate its explainability using SHAP, a well-established approach in the field of XAI. Comparative analysis demonstrates that GlassBoost outperforms SHAP in terms of precision, recall, and accuracy, with more balanced performance across the three metrics. Likewise, our review of literature findings indicate that Glassboost outperforms many other XAI models while retaining computational efficiency. In one of our configurations, GlassBoost achieved accuracy of 0.9868, recall of 0.9792, and precision of 0.9843 using only eight features within a tree structure of a maximum depth of four.

Context: Source code translation enables cross-platform compatibility, code reusability, legacy system migration, and developer collaboration. Numerous state-of-the-art techniques have emerged to address demand for efficient and accurate translation methodologies.Objective: This study compares code translation capabilities of Large Language Models (LLMs), specifically DeepSeek R1 and ChatGPT 4.1, evaluating their proficiency in translating code between programming languages. We systematically assess model outputs through quantitative and qualitative measures, focusing on translation accuracy, execution efficiency, and coding standard conformity. By examining each model's strengths and limitations, this work provides insights into their applicability for various translation scenarios and contributes to discourse on LLM potential in software engineering.Method: We evaluated translation quality from ChatGPT 4.1 and DeepSeek R1 using SonarQube Analyzer to identify strengths and weaknesses through comprehensive software metrics including translation accuracy, code quality, and clean code attributes. SonarQube's framework enables objective quantification of maintainability, reliability, technical debt, and code smells which are critical factors in software quality measurement. The protocol involved randomly sampling 500 code instances from 1695 Java programming problems. Java samples were translated to Python by both models, then analysed quantitatively using SonarQube metrics to evaluate adherence to software engineering best practices.Results: This comparative analysis reveals capabilities and limitations of state-of-the-art LLM-based translation systems, providing developers, researchers, and practitioners actionable guidance for model selection. Identified gaps highlight future research directions in automated code translation. Result s demonstrate that DeepSeek R1 consistently generates superior software quality compared to ChatGPT 4.1 across Sonar-Qube metrics.

Context The primary expectation from a software system revolves around its functionality. However, as the software development process advances, equal emphasis is placed on the quality of the software system for non-functional attributes like maintainability and performance. Tools are available to aid in this endeavour, assessing the quality of a software system from multiple perspectives.Objective This study aims to perform a comprehensive analysis of a particular set of source code analytical tools by examining diverse perspectives found in the literature and documentations. Given the vast array of programming languages available today, selecting appropriate source-code analytical tools presents a significant challenge. Therefore, this analysis aims to provide general insights to aid in selecting a more suitable analytical tool tailored to specific requirements.Method Seven prominent static analysis tools, namely SonarQube, Coverty, CodeSonar, Snyk Code, ESLint, Klocwork, and PMD, were chosen based on their prevalence in the literature and recognition in the software development community. To systematically categorise and organise their distinctive features and capabilities, a taxonomy was developed. This taxonomy covers crucial dimensions, including input support, technology employed, extensibility, user experience, rules, configurability, and supported languages.Results The comparative analysis highlights the distinctive strengths of each tool. SonarQube stands out as a comprehensive solution with a hybrid approach supporting static and dynamic code evaluations, accommodating multiple languages and integrating with popular Integrated Development Environments (IDEs). Coverity excels in identifying security vulnerabilities and defects, making it an excellent choice for security -focused development. CodeSonar prioritises code security and safety, offering a robust analysis. Snyk Code and ESLint, focusing on JavaScript, emphasise code quality and standards adherence. Klocwork is exceptional in defect detection and security analysis for C, C++, and Java. Lastly, PMD specialises in Java, emphasising code style and best practices.

In recent years, we have seen many examples of software products unintentionally causing demonstrable harm. Many guidelines for ethical and responsible computing have been developed in response. Dominant approaches typically attribute liability and blame to individual companies or actors, rather than understanding how the working practices, norms, and cultural understandings in the software industry contribute to such outcomes. In this paper, we propose an understanding of responsibility that is infrastructural, relational, and cultural; thus, providing a foundation to better enable responsible software engineering into the future. Our approach draws on Young's (2006) social connection model of responsibility and Star and Ruhleder's (1994) concept of infrastructure. By bringing these theories together we introduce a concept called infrastructural injustice, which offers a new way for software engineers to consider their opportunities for responsible action with respect to society and the planet. We illustrate the utility of this approach by applying it to an Open-Source software communities' development of Deepfake technology, to find key leverage points of responsibility that are relevant to both Deepfake technology and software engineering more broadly.

Hidden code clones negatively impact software maintenance, but manually detecting them in large codebases is impractical. Additionally, automated approaches find detection of syntactically-divergent clones very challenging. While recent deep neural networks (for example BERT-based artificial neural networks) seem more effective in detecting such clones, their pairwise comparison of every code pair in the target system(s) is inefficient and scales poorly on large codebases. We present SSCD, a BERT-based clone detection approach that targets high recall of Type 3 and Type 4 clones at a very large scale (in line with our industrial partner's requirements). It computes a representative embedding for each code fragment and finds similar fragments using a nearest neighbor search. Thus, SSCD avoids the pairwise-comparison bottleneck of other neural network approaches, while also using a parallel, GPU-accelerated search to tackle scalability. This article describes the approach, proposing and evaluating several refinements to improve Type 3/4 clone detection at scale. It provides a substantial empirical evaluation of the technique, including a speed/efficacy comparison of the approach against SourcererCC and Oreo, the only other neural-network approach currently capable of scaling to hundreds of millions of LOC. It also includes a large in-situ evaluation on our industrial collaborator's code base that assesses the original technique, the impact of the proposed refinements and illustrates the impact of incremental, active learning on its efficacy. We find that SSCD is significantly faster and more accurate than SourcererCC and Oreo. SAGA, a GPU-accelerated traditional clone detection approach, is a little better than SSCD for T1/T2 clones, but substantially worse for T3/T4 clones. Thus, SSCD is both scalable to industrial code sizes, and comparatively more accurate than existing approaches for difficult T3/T4 clone searching. In-situ evaluation on company datasets shows that SSCD outperforms the baseline approach (CCFinderX) for T3/T4 clones. Whitespace removal and active learning further improve SSCD effectiveness.

Software architecture assists developers in addressing non-functional requirements and in maintaining, debugging, and upgrading their software systems. Consequently, consistency between the designed architecture and the implemented software system itself is important; without this consistency the non-functional requirements targeted may not be addressed and architectural documentation may mis-direct maintenance efforts that target the associated code-base. But often, when software is initially implemented or subsequently evolved, the designed architecture and software architecture become inconsistent, with the implemented structure degraded due to issues like developer time-pressures, or ambiguous communication of the designed architecture. In such cases, Software Architecture Recovery (SAR) or consistency approaches can be applied to reconstruct the architecture of the software system and possibly to compare it to/re-align it with the designed architecture. Many SAR approaches have been proposed in the research. However, choosing an appropriate architecture recovery approach for software systems is still an open issue. Consequently, this research aims to conduct a tertiary-mapping study based on available secondary studies of architecture recovery approaches, to uncover important characteristics, towards the selection of appropriate SAR approaches. This research has aggregated 13 secondary studies and 10 primary studies beyond 2020 from 5 databases and, in doing so, identified 111 architecture recovery approaches. Based on these approaches, a taxonomy, containing nine main SAR-selection categories is proposed and a framework (in the form of a supporting tool to help developers select an appropriate SAR approach) has been developed. Finally, this research identifies six potential open research gaps related to the underlying research that could be helpful for guiding research in the future.

Microservices architecture has gained significant traction, in part owing to its potential to deliver scalable, robust, agile, and failure-resilient software products. Consequently, many companies that use large and complex software systems are actively looking for automated solutions to decompose their monolith applications into microservices. This paper rigorously examines 35 research papers selected from well-known databases using a Systematic Literature Review (SLR) protocol and snowballing method, extracting data to answer the research questions, and presents the following four contributions. First, the Monolith to Microservices Decomposition Framework (M2MDF) which identifies the major phases and key elements of decomposition. Second, a detailed analysis of existing decomposition approaches, tools and methods. Third, we identify the metrics and datasets used to evaluate and validate monolith to microservice decomposition processes. Fourth, we propose areas for future research. Overall, the findings suggest that monolith decomposition into microservices remains at an early stage and there is an absence of methods for combining static, dynamic, and evolutionary data. Insufficient tool support is also in evidence. Furthermore, standardised metrics, datasets, and baselines have yet to be established. These findings can assist practitioners seeking to understand the various dimensions of monolith decomposition and the community's current capabilities in that endeavour. The findings are also of value to researchers looking to identify areas to further extend research in the monolith decomposition space.

Large-scale source-code clone detection is a challenging task. In our previous work, we proposed an approach (SSCD) that leverages artificial neural networks and approximates nearest neighbour search to effectively and efficiently locate clones in large-scale bodies of code, in a time-efficient manner. However, our literature review suggests that the relative efficacy of differing neural network models has not been assessed in the context of large-scale clone detection approaches. In this work, we aim to assess several such models individually, in terms of their potential to maximize recall, while preserving a high level of precision during clone detection. We investigate if ensemble inference (in this case, using the results of more than one of these neural network models in combination) can further assist in this task. To assess this, we employed four state-of-the-art neural network models and evaluated them individually/in combination. The results, on an illustrative dataset of approximately 500K lines of C/C++ code, suggest that ensemble inference outperforms individual models in all trialled cases, when recall is concerned. Of individual models, the ADA model (belonging to the ChatGPT family of models) has the best performance. However commercial companies may not be prepared to hand their proprietary source code over to the cloud, as required by that approach. Consequently, they may be more interested in an ensemblecombination of CodeBERT-based and CodeT5 models, resulting in similar (if slightly lesser) recall and precision results.

BACKGROUND:Digital Contact Tracing is seen as a key tool in reducing the propagation of Covid-19. But it requires high uptake and continued participation across the population to be effective. To achieve sufficient uptake/participation, health authorities should address, and thus be aware of, user concerns.AIM:This work manually analyzes user reviews of the Irish Heath Service Executive's (HSE) Contact Tracker app, to identify user concerns and to lay the foundations for subsequent, large-scale, automated analyses of reviews. While this might seem tightly scoped to the Irish context, the HSE app provides the basis for apps in many jurisdictions in the USA and Europe.METHODS:Manual analysis of (1287) user reviews from the Google/Apple playstores was performed, to identify the aspects of the app that users focused on, and the positive/negative sentiment expressed.RESULTS:The findings suggest a largely positive sentiment towards the app, and that users thought it handled data protection and transparency aspects well. But feedback suggests that users would appreciate more targeted feedback on the incidence of the virus, and facilities for more proactive engagement, like notifications that prompt users to submit their health status daily. Finally, the analysis suggests that the "android battery" issue and the backward-compatibility issue with iPhones seriously impacted retention/uptake of the app respectively.CONCLUSION:The HSE have responded to the public's desire for targeted feedback in newer versions, but should consider increasing the app's proactive engagement. The results suggest they should also raise the backward compatibility issue, regarding older iPhones, with Apple.

Feature Location (FL) is a core software maintenance activity that aims to locate observable functionalities in the source code. Given its key role in software change, a vast array of Feature Location Techniques (FLTs) have been proposed but, as more and more FLTs are introduced, the selection of an appropriate FLT is an increasingly difficult problem. One consideration is the characteristics of the features being sought. For example, in the code associated with the feature, programmers may have named identifiers consistently, and with meaningful naming conventions, or not, and this may impact on the suitability of different FLTs. The suggestion that such characteristics matter has implicit support in the literature: An analysis of existing FLT empirical studies reveals that the system under study can often have a stronger impact on FLT performance than differing FLTs themselves. To understand this interaction between feature characteristics and FLTs better, this paper proposes a suite of feature-characteristic metrics that are postulated to control FLTs’ performance, holistically across FLTs and impacting on individual FLTs to different degrees. To evaluate the suite, a controlled experiment is performed, using 878 features, to probe the relationship between the metrics and the performance of four FTL techniques: three commonly-used techniques and one state-of-the-art technique. The evaluation is performed using four commonly used evaluation measures and extended by employing 41 other established source-code metrics as extraneous variables. Results of the empirical evaluation suggest that the feature-metric suite presented impacts FLT performance holistically, and impacts different FLTs to different degrees. Thus, this paper moves towards the more standard selection of appropriate FLTs, with respect to the prominent feature characteristics in the software systems under study, and more rigorous consideration of the features selected to compare FLTs.

Background In recent years, social media has become a major channel for health-related information in Saudi Arabia. Prior health informatics studies have suggested that a large proportion of health-related posts on social media are inaccurate. Given the subject matter and the scale of dissemination of such information, it is important to be able to automatically discriminate between accurate and inaccurate health-related posts in Arabic. Objective The first aim of this study is to generate a data set of generic health-related tweets in Arabic, labeled as either accurate or inaccurate health information. The second aim is to leverage this data set to train a state-of-the-art deep learning model for detecting the accuracy of health-related tweets in Arabic. In particular, this study aims to train and compare the performance of multiple deep learning models that use pretrained word embeddings and transformer language models. Methods We used 900 health-related tweets from a previously published data set extracted between July 15, 2019, and August 31, 2019. Furthermore, we applied a pretrained model to extract an additional 900 health-related tweets from a second data set collected specifically for this study between March 1, 2019, and April 15, 2019. The 1800 tweets were labeled by 2 physicians as accurate, inaccurate, or unsure. The physicians agreed on 43.3% (779/1800) of tweets, which were thus labeled as accurate or inaccurate. A total of 9 variations of the pretrained transformer language models were then trained and validated on 79.9% (623/779 tweets) of the data set and tested on 20% (156/779 tweets) of the data set. For comparison, we also trained a bidirectional long short-term memory model with 7 different pretrained word embeddings as the input layer on the same data set. The models were compared in terms of their accuracy, precision, recall, F1 score, and macroaverage of the F1 score. Results We constructed a data set of labeled tweets, 38% (296/779) of which were labeled as inaccurate health information, and 62% (483/779) of which were labeled as accurate health information. We suggest that this was highly efficacious as we did not include any tweets in which the physician annotators were unsure or in disagreement. Among the investigated deep learning models, the Transformer-based Model for Arabic Language Understanding version 0.2 (AraBERTv0.2)-large model was the most accurate, with an F1 score of 87%, followed by AraBERT version 2–large and AraBERTv0.2-base. Conclusions Our results indicate that the pretrained language model AraBERTv0.2 is the best model for classifying tweets as carrying either inaccurate or accurate health information. Future studies should consider applying ensemble learning to combine the best models as it may produce better results.

Objective This study aims to gather public opinion on the Irish “COVID Tracker” digital contact tracing (DCT) App, with particular focus on App usage, usability, usefulness, technological issues encountered, and potential changes to the App. Methods A 35-item online questionnaire was deployed for 10 days in October 2020, 3 months after the launch of the Irish DCT App. Results A total of 2889 completed responses were recorded, with 2553 (88%) respondents currently using the App. Although four in five users felt the App is easy to download, is easy to use and looks professional, 615 users (22%) felt it had slowed down their phone, and 757 (28%) felt it had a negative effect on battery life. Seventy-nine percent of respondents reported the App's main function is to aid contact tracing. Inclusion of national COVID-19 trends is a useful ancillary function according to 87% of respondents, and there was an appetite for more granular local data. Overall, 1265 (44%) respondents believed the App is helping the national effort, while 1089 (38%) were unsure. Conclusions DCT Apps may potentially augment traditional contact tracing methods. Despite some reports of negative effects on phone performance, just 7% of users who have tried the App have deleted it. Ancillary functionality, such as up-to-date regional COVID-19, may encourage DCT App use. This study describes general positivity toward the Irish COVID Tracker App among users but also highlights the need for transparency on effectiveness of App-enabled contact tracing and for study of non-users to better establish barriers to use.

Background The silent transmission of COVID-19 has led to an exponential growth of fatal infections. With over 4 million deaths worldwide, the need to control and stem transmission has never been more critical. New COVID-19 vaccines offer hope. However, administration timelines, long-term protection, and effectiveness against potential variants are still unknown. In this context, contact tracing and digital contact tracing apps (CTAs) continue to offer a mechanism to help contain transmission, keep people safe, and help kickstart economies. However, CTAs must address a wide range of often conflicting concerns, which make their development/evolution complex. For example, the app must preserve citizens’ privacy while gleaning their close contacts and as much epidemiological information as possible. Objective In this study, we derived a compare-and-contrast evaluative framework for CTAs that integrates and expands upon existing works in this domain, with a particular focus on citizen adoption; we call this framework the Citizen-Focused Compare-and-Contrast Evaluation Framework (C3EF) for CTAs. Methods The framework was derived using an iterative approach. First, we reviewed the literature on CTAs and mobile health app evaluations, from which we derived a preliminary set of attributes and organizing pillars. These attributes and the probing questions that we formulated were iteratively validated, augmented, and refined by applying the provisional framework against a selection of CTAs. Each framework pillar was then subjected to internal cross-team scrutiny, where domain experts cross-checked sufficiency, relevancy, specificity, and nonredundancy of the attributes, and their organization in pillars. The consolidated framework was further validated on the selected CTAs to create a finalized version of C3EF for CTAs, which we offer in this paper. Results The final framework presents seven pillars exploring issues related to CTA design, adoption, and use: (General) Characteristics, Usability, Data Protection, Effectiveness, Transparency, Technical Performance, and Citizen Autonomy. The pillars encompass attributes, subattributes, and a set of illustrative questions (with associated example answers) to support app design, evaluation, and evolution. An online version of the framework has been made available to developers, health authorities, and others interested in assessing CTAs. Conclusions Our CTA framework provides a holistic compare-and-contrast tool that supports the work of decision-makers in the development and evolution of CTAs for citizens. This framework supports reflection on design decisions to better understand and optimize the design compromises in play when evolving current CTAs for increased public adoption. We intend this framework to serve as a foundation for other researchers to build on and extend as the technology matures and new CTAs become available.