The rapid expansion of online programming exercise platforms has brought abundant learning resources for programming education, but also presents challenges for personalized exercise recommendation due to missing or imprecise knowledge annotations and the sparsity of learner-exercise interaction records, which together lead to reduced accuracy and a lack of explainability in the recommendation results. Existing natural language processing and large language models (LLM)-based annotation methods struggle to capture implicit knowledge and often generate redundant or inconsistent results. Moreover, mainstream recommendation systems are ineffective at handling the severe sparsity of interaction sequences in programming exercise datasets, and typically model student preferences in a single dimension—overlooking key educational factors such as knowledge gaps, difficulty tolerance, and preferred learning rhythm. To address these challenges, we propose KDG-Rec, a novel framework that first introduces AgentCo-KAS, a multiagent LLM-based collaborative annotation method with role-specific fine-tuning and cross-agent verification for high-precision, fine-grained knowledge annotation. Building on these enriched annotations, we develop a disentangled graph neural network model that constructs dual exercise-interaction graphs to effectively capture learning patterns at multiple granularities in interaction sequences and explicitly decouples student preferences into four interpretable dimensions for adaptive fusion. Extensive experiments on real-world datasets demonstrate that KDG-Rec outperforms nine state-of-the-art methods across multiple metrics, significantly advancing personalized programming exercise recommendation.

In recent times, programming knowledge tracing, as a fundamental task in programming learning analytics, has been widely applied in education, enabling accurate diagnosis of learners' mastery of programming knowledge concepts across diverse learning contexts. Existing programming knowledge tracing methods often overlook two key challenges: knowledge concept annotation deficiencies caused by the multiple-solution nature of programming problems and inadequate modeling of the actual progression of knowledge states within students' continuous submission sequences for targeted problems. Consequently, the information contained in the submitted code is underutilized, negatively impacting the estimation of students' evolving knowledge states. In this paper, we propose HGAKT-Code, a code-aware knowledge tracing framework for programming education assisted by Online Judge (OJ) systems. Our model integrates three novel components: (1) a Hierarchical Multi-label Code Classification (HMCC) algorithm to mitigate Q-matrix inaccuracies caused by diverse code solutions; (2) a code similarity-enhanced attention mechanism to better capture critical submissions reflecting subtle changes in students' knowledge states; and (3) a heterogeneous GNN with designed graph attention, modeling both conceptual and temporal learning dynamics. We evaluated HGAKT-Code on three OJ datasets and achieve AUC improvements of 4.2%, 3.0%(public OJ dataset) and 2.9% (institutional OJ dataset) over state-of-the-art baselines, respectively, demonstrating its effectiveness across educational settings.

Software development effort estimation (SDEE) is essential for effective project planning and relies heavily on data quality affected by incomplete datasets. Missing data (MD) are a prevalent problem in machine learning, yet many models treat it arbitrarily despite its significance. Inadequate handling of MD may introduce bias into the induced knowledge. It can be challenging to choose optimal imputation approaches for software development projects. This article presents a novel incomplete value imputation model (NIVIM) that uses a variational autoencoder (VAE) for imputation and synthetic data. By combining contextual and resemblance components, our approach creates an SDEE dataset and improves the data quality using contextual imputation. The key feature of the proposed model is its applicability to a wide variety of datasets as a preprocessing unit. Comparative evaluations demonstrate that NIVIM outperforms existing models such as VAE, generative adversarial imputation network (GAIN), k$$ k $$-nearest neighbor (K-NN), and multivariate imputation by chained equations (MICE). Our proposed model NIVIM produces statistically substantial improvements on six benchmark datasets, that is, ISBSG, Albrecht, COCOMO81, Desharnais, NASA, and UCP, with an average improvement in RMSE of 11.05% to 17.72% and MAE of 9.62% to 21.96%.

Identifying interaction scenarios between a system and its actors from the high-level requirements and forming use case basic flows is crucial in requirement refinement. Traditional manual methods often yield incomplete or inaccurate flows due to engineers' limited domain expertise, while rule-based methods-relying on predefined parsing rules-suffer from linguistic ambiguities and domain-dependent limitations. Although large language model (LLM) approaches leverage rich domain knowledge and robust natural language processing, they are constrained by input length, generation instability, and the risk of out-of-system outputs, frequently resulting in context-unaware or irrelevant flows. To overcome these challenges, this paper proposes BFGen to generate context-compliant basic flows strictly adhering to domain constraints and requirement boundaries. BFGen employs LLMs to accurately extract domain-specific terms and interactions, and integrates a Relational Graph Attention Network with attention preservation factors to model logical dependencies and domain constraints effectively. Empirical evaluations on 13 public and 7 industrial datasets show that BFGen outperforms leading baselines by approximate to 14% in Precision, approximate to 7-25% in Recall, approximate to 11-30% in F1 Score, and approximate to 10-19% in AUC. Furthermore, our evaluations confirm the effectiveness of both the LLM module and the attention preservation factors, and assess the impact of requirement completeness on the performance of BFGen.

Sequential recommendation represents a well-explored yet challenging domain within research. Despite significant advancements in GNN-based methods for modeling intricate patterns in user-item interaction sequences, these methods face difficulties in capturing nuanced semantics in sequences with sparse dependencies and noise, and often struggle with short sequences. Additionally, distinguishing higher-order semantic distinctions among diverse user interests is still challenging, and existing GNN-based methods can be computationally intensive. To address these challenges, we propose a Dual-Graph approach for Sequential Recommendation, DGSR. We construct individual interaction graphs for each sequence, and a heterogeneous global interaction graph that incorporates user identity as an attribute of user edges. Last-item augmented GGNN is employed within individual interaction graphs to mitigate the impact of sparse dependencies and noise, thereby extracting the most recent interests for each user more effectively. Moreover, within the global graph, we propose a parameter-efficient heterogeneous GNN to extract high-order interest distinctions among diverse users while maintaining low computational complexity. Finally, we utilize vanilla transform mechanism to integrate intra- and inter-user interests from both types of graphs. Experiments on four publicly available datasets demonstrate that our method achieves state-of-the-art performance, surpassing all baseline methods.

This investigation focuses on refining software effort estimation (SEE) to enhance project outcomes amidst the rapid evolution of the software industry. Accurate estimation is a cornerstone of project success, crucial for avoiding budget overruns and minimizing the risk of project failures. The framework proposed in this article addresses three significant issues that are critical for accurate estimation: dealing with missing or inadequate data, selecting key features, and improving the software effort model. Our proposed framework incorporates three methods: the Novel Incomplete Value Imputation Model (NIVIM), a hybrid model using Correlation-based Feature Selection with a meta-heuristic algorithm (CFS-Meta), and the Heterogeneous Ensemble Model (HEM). The combined framework synergistically enhances the robustness and accuracy of SEE by effectively handling missing data, optimizing feature selection, and integrating diverse predictive models for superior performance across varying project scenarios. The framework significantly reduces imputation and feature selection overhead, while the ensemble approach optimizes model performance through dynamic weighting and meta-learning. This results in lower mean absolute error (MAE) and reduced computational complexity, making it more effective for diverse software datasets. NIVIM is engineered to address incomplete datasets prevalent in SEE. By integrating a synthetic data methodology through a Variational Auto-Encoder (VAE), the model incorporates both contextual relevance and intrinsic project features, significantly enhancing estimation precision. Comparative analyses reveal that NIVIM surpasses existing models such as VAE, GAIN, K-NN, and MICE, achieving statistically significant improvements across six benchmark datasets, with average RMSE improvements ranging from 11.05% to 17.72% and MAE improvements from 9.62% to 21.96%. Our proposed method, CFS-Meta, balances global optimization with local search techniques, substantially enhancing predictive capabilities. The proposed CFS-Meta model was compared to single and hybrid feature selection models to assess its efficiency, demonstrating up to a 25.61% reduction in MSE. Additionally, the proposed CFS-Meta achieves a 10% (MAE) improvement against the hybrid PSO-SA model, an 11.38% (MAE) improvement compared to the Hybrid ABC-SA model, and 12.42% and 12.703% (MAE) improvements compared to the hybrid Tabu-GA and hybrid ACO-COA models, respectively. Our third method proposes an ensemble effort estimation (EEE) model that amalgamates diverse standalone models through a Dynamic Weight Adjustment-stacked combination (DWSC) rule. Tested against international benchmarks and industry datasets, the HEM method has improved the standalone model by an average of 21.8% (Pred()) and the homogeneous ensemble model by 15% (Pred()). This comprehensive methodology underscores our model’s contributions to advancing software project management (SPM) through advanced predictive modeling, setting a new benchmark for software engineering effort estimation.

Application Programming Interfaces (APIs) have become common in contemporary software development. Many automated API recommendation methods have been proposed. However, these methods suffer from a deficit of using domain knowledge, giving rise to challenges like the “cold start” and “semantic gap” problems. Consequently, they are unsuitable for test function recommendation, which recommends test functions for test engineers to implement test cases formed with various test steps. This paper introduces an approach named TOKTER, which recommends test functions leveraging test-oriented knowledge graphs. Such a graph contains domain concepts and their relationships related to the system under test and the test harness, which is constructed from the corpus data of the concerned test project. TOKTER harnesses the semantic associations between test steps (or queries) and test functions by considering literal descriptions, test function parameters, and historical data. We evaluated TOKTER with an industrial dataset and compared it with three state-of-the-art approaches. Results show that TOKTER significantly outperformed the baseline by margins of at least 36.6% in mean average precision (MAP), 19.6% in mean reciprocal rank (MRR), and 1.9% in mean recall (MR) for the top-10 recommendations.

In practices, test cases are often designed by test engineers based on the functionalities of the System under Test (SUT) in parallel and independently. This can lead to a lack of a comprehensive overview of the test architecture, hampering the reuse of test functions when implementing new test cases. To address this challenge, we propose ATAG, an automated test architecture generation approach, which employs an optimization algorithm to retrieve highly cohesive and loosely coupled test functions based on control flows and data flows of test cases. We also equip ATAG with a newly proposed BERT-based model, i.e., FunBERT, for generating test function names. We conducted an empirical study with three industrial datasets to evaluate the effectiveness of ATAG and FunBERT. Results show that test architectures generated with ATAG (benefiting from both control and data flows) improved, on average, ≈ 26–35 ≈ 28–50

To achieve the primary teaching goal of a programming course, the typical teaching method is to improve students' programming skills by assigning them a number of programming tasks and to check the quality of programs finished by students by using online judge system. Currently, these judge systems mainly contribute to evaluating how well a program satisfies its task requirement. Besides this basic role, a program judge system should also help students to progressively improve their programs' quality throughout the entire development lifecycle, rather than simply assessing the correctness of their programs. This assistance encompasses program construction, testing, and bug fixing stages. To meet the diverse needs of program evaluation and handle the high workload generated by a large number of evaluation tasks, this paper proposes a distributed microservice based program judge system called FLOD (Full Lifecycle Online juDge) built on time driven interactive judge cores, together with the configurable testing services. FLOD provides teachers with configurable judge workflow and running environment, upon which various programming tasks can be assigned and implemented. FLOD can automatically access the code repositories of students by integrating with Gitlab and provide interactive code judge services to feedback students instantly with the bugs and bad code styles found such that students can improve their programs at their own paces. FLOD has been successfully deployed to serve the Object Oriented Design and Construction (OODC) course offered in Beihang university since 2019. To assess its effectiveness, we propose the metrics and collect the usage data from 2019 to 2022. The results show that FLOD effectively supports the accompanying programming teaching with considerable reliability and scalability, which is very important to achieve the teaching goal for our course with over 300 students.

The software industry has experienced rapid expansion in recent years, with software development now essential to the success of many multinational corporations. The demand for complex software systems has dramatically increased, effective software development has become crucial, given the limitations of resources such as money, time, and labor. Cost and effort calculations significantly impact the development process and client needs, and project failure is often caused by errors in job estimating. Underestimating a project’s cost and effort can have severe repercussions, such as exceeding the project’s budget. Project overruns, on the other hand, can also have a detrimental impact on software projects’ successful completion. Researchers and experts in the software industry are continually exploring ways to keep management and development productivity at high levels. However, standalone estimating models have revealed inadequacies over the last decade, and they have not produced any noteworthy research results. Recent literature suggests that opting for ensemble models would yield better results than standalone models. We have proposed a heterogeneous ensemble effort estimation (EEE) model in this research. Our proposed model comprises standalone estimating models such as Use Case Point, Expert Judgment (EJ), and Artificial Neural Network (ANN). We combined the effort of each unique base model using linear combination rule. To validate our model’s effectiveness, we applied it to the benchmark dataset, the International Software Benchmarking Standards Group (ISBSG), using three different variations to avoid biases. We further applied the trained models to industry use cases for cross-validation. Our study’s findings demonstrated that, in comparison to stand-alone estimate strategies, the ensemble technique produced better estimation results. Finally, our study proposes a heterogeneous ensemble effort estimation model that outperforms standalone models in terms of accuracy. This model has the potential to aid in effective software development, particularly in project cost and effort estimation.

Regression testing is required to assure the quality of each iteration of microservice systems. Test case selection is one of main techniques to optimize regression testing. Existing techniques mainly involve artifacts acquisition, processing and maintenance, thus hard to apply in microservice regression testing since it is difficult to obtain and process required artifacts from multiple development teams, which is normal in cases of microservice systems. This paper proposes a novel approach, namely MRTS-BP, which takes API gateway logs instead of artifacts as inputs. By mining service dependencies from API gateway logs, MRTS-BP analyzes service change impacts based on a propagation calculation, and selects test cases affected by changes based on impact degree values. To evaluate the effectiveness of MRTS-BP, empirical studies based on four real deployed systems are presented. Retest-all strategy and a regression testing selection approach based on control flow graphs called RTS-CFG are compared with MRTS-BP. The results show that, MRTS-BP can significantly reduce both the number of test cases and overall time cost while maintaining the fault detection capability of selected test suite, and that MRTS-BP can save more time cost than RTS-CFG with the similar safety and precision.

Implementing test cases as programs to automate test execution is a popular testing practice. Current industrial practices usually use test functions to implement the test steps of a test case and then to compose the executable test case by choosing the test functions to call manually. It is time-consuming and could lead to invalid test results by selecting inappropriate test functions. In this article, we propose an automatic test function recommendation approach named Scenario-based Recommendation of TEst Function (SRTEF). Given a test step of a test case, SRTEF uses the weighted description similarity and the scenario similarity to recommend test functions. The description similarity utilizes the deep structured semantic model (DSSM) to measure the relatedness between a test step and a test function by their literal descriptions. The test scenario and the test function usage scenario are considered to calculate the scenario similarity. SRTEF has been successfully applied in Huawei. The systematic experiments have been conducted to evaluate SRTEF by using the dataset from Huawei and comparing with BiInformation source-based KnowledgE Recommendation (BIKER), reported as the best approach so far. The results show that SRTEF outperforms BIKER with significant positive ratios consistently in all the three selection strategies, i.e., Top-3, Top-5, and Top-10. The DSSM shows its advantage over word embedding by the double performance of capturing the semantic relatedness in SRTEF.

Regression testing is required in each development iteration of microservice systems. Test case prioritization, which improves the fault detection rate by optimizing the execution order of test cases, is one of the main techniques to optimize regression testing. Existing test case prioritization techniques mainly rely on artifacts such as codes and system models, which are limited to microservice systems with service autonomy, development method diversity, and large service scale. This paper proposes a test case prioritization approach based on service ranking referred to as TCP-SR. TCP-SR ranks the services based on API gateway logs. The weights of test cases are calculated with the result of service ranking, which could be used to order test cases with single-objective and multi-objective strategies. To evaluate the effectiveness of TCP-SR, the empirical study based on four microservice systems is presented. The results show that the fault detection rate of TCP-SR is almost twice as high as that of the random prioritization technique, and almost the same as the prioritization technique based on WS-BPEL but requires much less prioritization time cost.

In recent years, Convolutional Neural Network (CNN) has achieved a great success in computer vision. However, at present, for an image classification task, there is no CNN model that can perform 100% accurately due to insufficient or excessive feature learning. Once a CNN model deployed to perform tasks online, misclassified samples might lead the system with the CNN model deployed to enter an unsafe state such as collisions. To assess the performance of such online models, we, in this paper, propose Parallel Signal Routing Paths (PSRP) method to identify misclassified samples by extracting execution paths for each sample and comparing inherent feature differences in terms of CNN nodes between misclassified and well-classified samples, for the ultimate aim of addressing the challenge of test data not having ground-truth labels in online environment where the CNN models are deployed, and give availability results for applying PSRP on 3 public datasets and 3 typical CNN models.

微服务系统每轮迭代过程中都需要进行回归测试,大量重复测试会造成资源浪费,可通过减少测试用例集的规模来降低成本,以提高测试效率.现有测试用例集缩减技术主要依赖系统规约和架构描述作为输入,对于具有服务自治、调用关系不确定等特点的微服务系统实用性受限.并且,现有测试用例集缩减技术很少考虑使用场景,测试用例集很难反映用户关切.提出了 一种基于API网关层日志挖掘的测试用例集缩减技术,从API网关层日志中挖掘出能够反映服务使用场景的频繁调用路径,建立频繁路径与测试用例的关联关系,进而构建搜索图,并基于启发式搜索生成测试用例缩减集.描述了该技术的完整过程,并基于一个集成办公微服务系统进行了实验.实验结果表明:该技术能够缩减测试用例规模40％以上,且缺陷发现能力降幅不超过10％.

Regression testing is the optimal technique that can be used in each iteration of microservice systems. However, regression testing prioritization is the only main method that gives better results. These techniques directly involve the processes of artifacts, data acquisition, analysis, and maintenance. The microservice systems have input data, which are difficult to obtain and control, while such processes are of high costs with impractical design. This paper gives a detailed study on testing prioritization technique, which is referred to as CIPC. As there are dependencies between services from API gateway logs, a novel CIPC algorithm is proposed, which is based on belief propagation. There are some rules that are directly affected by service changes. Therefore, the higher execution order of test case prioritizes CIPC, which is based on impact changes. Multiobjective prioritization algorithm is based on heuristic searching, in which sequence test cases are done by coverage. By evaluating the effectiveness of CIPC, the empirical study presents five microservice systems and four different techniques. The results describe that CIPC has improved fault detection rate with acceptable time and cost. The technique is more practical than typical artifacts, which are based on increments of system scales.

Machine learning (ML) system is highly data-dependent. It turns out different behavior and performance by using different training data. Since its behavior and performance are more unpredictable than traditional software. The requirement analysis of ML system should focus on model development and its training. This paper reveals how the model development and training are conducted by reviewing the published papers. Based on the principles by Kitchenham, we propose a concern driven systematic literature review method and choose face recognition facilitated ML system as case study. We identify the concerns and then research questions from data perspective, and answer the questions based on the collected literature data. The results show that the current studies included in review have already recognized the role of data played in ML system. Since there is no standard template and method to report data and the related processing procedure, the content and detail about the data reported vary from study to study and are not systematic, which is a great challenge to both practitioners and users.

Implementing test cases to automate test execution is a popular testing practice currently. A stepwise test case consists of several sequential test steps. Given a test function library, the typical way to implement a test case is calling the existing test functions in the library to reduce test cost. How to find the appropriate test function(s) to implement a test step in a given test case thus becomes an important problem. However, in current testing practices, test engineers usually select the appropriate test function manually by experience. It is time-consuming and could lead to invalid test results by selecting inappropriate or wrong test functions to call. In this paper, we propose an automatic test function recommendation approach with scenario named SRTEF (Scenario-based Recommendation of TEst Function). Given a test step, SRTEF uses two levels of similarities to recommend test functions, description similarity and scenario similarity. The description similarity measures the semantic relatedness between the test step and test function by their literal descriptions. To calculate the scenario similarity, SRTEF at first retrieves a set of historical test cases that contains test step(s) semantically similar to the given test step; then the scenario similarity between test step and test function is calculated according to the calling relation between retrieved test case and test function, and the co-occurrence relation among test functions. SRTEF has been successfully applied in Huawei. We evaluate SRTEF by using the dataset from Huawei and comparing with BIKER, reported as the best recommendation approach so far. The results show that SRTEF outperforms the BIKER approach by at least 49% in Mean Average Precision, 33% in Mean Reciprocal Rank, and 25% in Mean Recall.

As one mainstream of current software development, Object-oriented programming has become one key course for undergraduate students in Computer Science. Since Object-oriented concepts are difficult to understand for students, small programming exercises are used to train and help the students, and the study performance is evaluated based on the quality of the submitted source code. The common practice of code assess-ment in programming courses is checking whether the submitted projects pass carefully-designed test cases. However, even some projects pass all test cases, they may have bad software design and do not use the knowledge of Object-oriented programming well, especially in the early stage of courses. Therefore, we propose an anomaly detection approach for early warning in Object-oriented programming courses, which can automatically find the abnormal application of Object-oriented knowledge. In our approach, we conduct static analysis on the code submitted by students. Typical Objected-oriented metrics are extracted, and students are divided into two groups by K-means clustering: being good at Object-objected knowledge or not, and finally detect anomalous students based on the distance from cluster centers. We evaluate our approach on the realistic data sets collected from our Object-oriented programming course, and experimental results show the effectiveness of our method.

This pilot study examines how students' performance has evolved in an Object-oriented (OO) programming course and contributes to the learning analytic framework for similar programming courses in university curriculum. First, we briefly introduce the research background, a novel OO teaching practice with consecutive and iterative assignments consisting of programming and testing assignments. We propose a planned quantitative method for assessing students' gains in terms of programming performance and testing performance. Based on real data collected from students who engaged in our course, we use trend analysis to observe how students' performance has improved over the whole semester. By using correlation analysis, we obtain some interesting findings on how students' programming performance correlates with testing performance, which provides persuasive empirical evidence in integrating software testing practices into an Object-oriented programming curriculum. Then, we conduct an empirical study on how students' design competencies are represented by their program code quality changes over consecutive assignments by analyzing their submitted source code in the course system and the GitLab repository. Three different kinds of profiles are found in the students' program quality in the OO design level. The group analysis results reveal several significant differences in their programming performance and testing performance. Moreover, we conduct systematical explanations on how students' programming skill improvement can be attributed to their object-oriented design competency. By performing principal component analysis on software statistical data, a predictive OO metrics suite for both students' programming performance and their testing performance is proposed. The results show that these quality factors can serve as useful predictors of students' learning performance and can provide effective feedback to the instructors in the teaching practices.