Policymakers and researchers collect data on student and school socio-economic background for various purposes, such as prioritising students from socio-economically disadvantaged backgrounds for additional resources, understanding the background characteristics associated with educational achievement, or monitoring the progress of policy reforms and initiatives. International large-scale assessments gather detailed information from participating students and school principals and use this to calculate composite measures of socio-economic status. At the same time, policymakers usually have access to national socio-economic indicators based on administrative data. Given the impact of socio-economic indicators on monitoring and shaping education policy, key issues related to the convergent and criterion-related validity of the international and national socio-economic measures are of concern to both policymakers and researchers. In this paper, we compare socio-economic indicators from the 2019 Trends in International Mathematics and Science Study and the 2022 Programme for International Student Assessment with two key indicators used in policy-making in Ireland (the Pobal HP Deprivation Index and school participation in the Delivering Equality of Opportunity in Schools programme) and examine their associations. Furthermore, we consider the extent to which the indicators explain variance in mathematics achievement. Analyses include correlations, linear and logistic regression, and multilevel modelling. Findings show strong associations between the socio-economic indicators examined, with stronger associations among indicators derived from aggregated student-level data (r ≈ 0.7) than among inherently school-level indicators (r ≈ 0.6). Principals’ judgements about the proportion of disadvantaged students in their schools, collected via TIMSS and PISA, are found to be quite accurate insofar as these estimates align well with both national indicators and other ILSA-derived aggregated indicators. International indicators were more strongly predictive of achievement in mathematics than the national indicators, with the national indicators explaining little additional variance in TIMSS and PISA mathematics outcomes beyond the ILSA-derived SES indicators. This paper provides valuable empirical validity evidence for both international, and Ireland’s national, socio-economic indicators. The methodological approach presented here may also be valuable as a model for other countries participating in international large-scale assessments, particularly those that use their own administrative socio-economic data for policy-related decision-making.
Academic resilience—students’ capacity to succeed despite socioeconomic disadvantage—is widely studied, but little consensus exists on how it should be operationalized. Inconsistent definitions make it difficult to compare findings across studies and to draw conclusions about what promotes resilience. The present study asks whether different operationalizations and thresholds lead to different estimates of resilience and different conclusions about its predictors. Using PISA 2018 data from 75 education systems and based on an existing measurement framework, we compared three ways of operationalizing academic resilience: identifying high-achieving disadvantaged students, identifying students who outperform expectations given their socioeconomic status, and predicting achievement within a disadvantaged sample. We applied each at three relative threshold levels of disadvantage and/or high achievement, and assessed the consequences for the estimated prevalence of resilient students and for the associations of four factors (gender, self-perceived reading competence, reading enjoyment, attitude toward learning) with resilience. Prevalence estimates varied nearly fivefold across operationalizations, and rankings of education systems differed substantially. When associations were pooled meta-analytically, they were broadly consistent across operationalizations and threshold levels. When education systems were analyzed separately, however—as is standard in PISA-based research—findings were consistent in only 20–72
Abstract Background About half of the countries participating in TIMSS 2019 transitioned from paper-based to digital assessment in that cycle, with most remaining countries completing the transition in TIMSS 2023. All countries transitioning in 2019 completed a paper-based ‘bridge’ study to examine mode effects (differences in achievement related to the mode of administration) and ensure the comparability of scores from previous cycles. However, only four countries opted to administer a national comparison (bridge) study in 2023: Ireland, Azerbaijan, Bahrain, and Flanders (Belgium). This Brief Report documents findings from these four national comparison studies. The aim is to verify the extent to which the transition to digital testing may have affected student performance in these countries, noting that ten countries in TIMSS 2019 were previously found to have displayed country-level mode effects in at least one subject or grade. Findings We find no evidence of significant differences in overall performance by mode of administration, in either mathematics or science, in any of the four included countries. We also find no evidence of significant mode effects among boys or girls in Ireland, Flanders, or Bahrain. In Azerbaijan, boys achieved a significantly higher science score on the digital version of TIMSS compared to the paper version. Some differences in achievement by mode are observed at the level of the mathematics and science content and cognitive domains. These are limited and present no distinctive pattern in three of the countries studied although, again, more consistent differences are noted in Azerbaijan. Finally, some preliminary evidence is presented of differences in student performance on the digital assessment related to students’ access to a computer at home in Flanders, Bahrain, and Azerbaijan. Conclusions The findings provided here confirm that achievement estimates in TIMSS 2023 in these four countries were substantively unaffected at a high level by the change in the mode of administration. This should strengthen confidence in the use of these data for policymaking, educational practice, and further research. In particular, trend data from 2019 to 2023 in these countries can be more confidently interpreted as reflecting substantive trends or changes in achievement, giving researchers greater confidence that trends have not been significantly confounded by the procedural change of assessment mode. Evidence of some specific differences in each country could be considered further by national experts in curriculum and assessment. In particular, several differences have been observed that merit further consideration by researchers and policymakers in Azerbaijan.
Abstract There is growing interest in how young people’s socio-emotional skills, particularly motivation, shape educational outcomes. Yet less is known about how adolescents’ general achievement motivation and their specific aspirations to attend university relate to performance in high-stakes examinations. This paper provides new evidence using linked survey-administrative data from England collected shortly before students take high-stakes compulsory national exams. We distinguish between achievement motivation, measured using validated PISA scales, and university aspirations, defined by students’ stated application intentions. Our findings reveal substantial gender and immigrant-background gaps in university aspirations, with mixed patterns by socioeconomic status. We also document strong associations between motivation, aspirations and achievement; students who report high achievement motivation and aspire to elite universities score around 0.37 standard deviations higher in their compulsory exams than similar peers with lower motivation and no university aspirations, even after controlling for prior attainment and school fixed effects. These results highlight the distinct roles of motivation and university aspirations in shaping educational inequality.
Abstract This study examined associations between school digital conditions and students’ computer and information literacy (CIL) in 12 ICILS 2023 education systems that administered the optional ChatGPT questionnaire. The analysis used student achievement and questionnaire data linked with principal and ICT coordinator reports for 24,273 students in 1,049 schools. System-specific weighted two-level models were estimated for five CIL plausible values, adjusting for student sex, home internet access, computer experience, within-school socioeconomic background, and school socioeconomic composition. Five school-level conditions were examined jointly: positive ChatGPT expectations, negative ChatGPT expectations, pedagogical ICT hindrances, instructional ICT-use expectations, and ICT collaboration expectations. False-discovery-rate control was applied to the 60 primary focal tests, and a secondary equal-system-weight common-slope model and prespecified sensitivity analyses were used to assess the stability of the findings. None of the five school conditions produced an FDR-retained association with CIL. Three system-specific coefficients had raw p values below 0.05: ICT collaboration expectations were negative in Korea and positive in the Slovak Republic, and pedagogical ICT hindrances were negative in Korea; all had q = 0.730. The secondary common-slope model yielded confidence intervals including zero for all five focal predictors, and sensitivity analyses did not materially alter the primary conclusion. In contrast, student socioeconomic background and computer experience showed more consistent positive adjusted associations with CIL. Overall, the findings support cautious interpretation of pooled summaries alongside system-specific estimates and indicate no single common direct association between the measured school digital conditions and student CIL across the participating systems.
Abstract Background Mathematics achievement is associated with interacting individual and contextual factors, but these associations are often examined separately. As a result, existing research may overlook how affective characteristics combine within students and how these configurations are distributed across school settings. Objectives This study aimed to investigate the multilevel predictors of mathematics achievement by integrating machine learning and latent variable approaches using PISA 2022 data from nine European countries. Methods The analysis was conducted in a sequence of stages, summarised in Fig. 1. First, predictive models were used to identify the most important predictors of mathematics achievement. Second, Latent Profile Analysis was performed to classify students based on key socioeconomic and affective indicators. Third, a two-stage multilevel approach—a separate latent profile analysis of school-level indicators, linked to the student profiles through mixed-effects models—was used to identify school climate profiles and examine cross-level associations between school contexts and student profile membership. Results and conclusions Tree-based algorithms outperformed regularised linear models, indicating that mathematics achievement was characterised by complex and non-linear relationships among predictors. Socioeconomic resources and mathematics-related affective beliefs emerged as the strongest predictors. The analyses identified six student profiles and three school climate profiles, with marked differences in achievement across student groups. Student profiles were not randomly distributed across schools; instead, they were systematically associated with distinct school climates. These findings suggest that mathematics achievement is associated with synergistic combinations of individual and contextual factors, highlighting the importance of personalised interventions and equity-oriented policies that account for school-level selectivity.
Abstract Academic resilience refers to the ability of socioeconomically disadvantaged students to achieve high levels of academic performance despite adversity. This concept highlights students who succeed against the odds and provides insight into how education systems can better support equity and excellence. Drawing on data from TIMSS 2023 across 47 education systems, this study examines resilience in mathematics and science, with particular attention to subject- and gender-specific patterns. Socioeconomically disadvantaged students are identified as those in the bottom quartile of home educational resources distribution within each education system. Within this group, resilience students are identified using two commonly applied criteria: a fixed threshold using the TIMSS High International Benchmarks of 550 points, and a relative threshold identifying students who achieve in the top quartile of the national achievement distribution. The findings show that resilience patterns are partly definition-dependent, and underline the importance of examining resilience separately by subject and gender. Threshold-based indicators should be interpreted as descriptive measures of achievement among disadvantaged students rather than as comprehensive indicators of adaptive capacity or education-system effectiveness.
Abstract Background The recently introduced highly adaptive testing (HAT) design allows for a transparent and straightforward construction of the testing design of the Programme for International Student Assessment (PISA). It outperformed the multistage testing (MST) design used in PISA 2018 regarding test information and RMSE of individual ability estimates, but it is yet unknown whether it can also promote the estimation of population statistics which are used for reporting PISA results. The present paper thus examines the impact of HAT on plausible value-based estimates of population statistics (country means, standard deviations, proportions of students on proficiency levels) and the error variance components associated with these estimates. Methods The study involved simulating the PISA 2018 Reading assessment under both the MST design used in PISA 2018 and the HAT design across 18 OECD countries ( N = 110,958 students), utilizing the publicly available assessment and background data, and the official PISA item pools for Reading, Mathematics, and Science. Data generation, item administration, and population modeling, including the drawing of the PVs, followed the official PISA 2018 procedures. Results The HAT design significantly reduced the mean imputation variance of the country-specific mean reading proficiency by 19%, compared to the MST design. With regard to the proportions on the proficiency levels, the HAT design led to significant reductions in the imputation variance (− 8 to − 13%) and the total error variance (up to − 9%) compared to the MST design. In most countries and on average, the proportions on the proficiency Levels V and VI were significantly smaller for the HAT design; this indicates an overestimation of these proportions with the MST design. Conclusions The HAT significantly increases the measurement precision of population statistics. Moreover, it seems to make a more nuanced categorizations of individuals into proficiency levels possible. These results on population level illustrate the importance of using the best possible design. This will become even more important if future PISA assessments utilize improved stratification schemes that reduce the sampling variance, as the influence of the design on the precision of the reported population statistics will increase.
Abstract Background Digital reading has become an essential component of contemporary literacy, yet questions remain about whether digital reading assessments measure a construct distinct from traditional paper-based reading. This study examined whether variance unique to ePIRLS reflects a distinct digital-reading construct or is better explained by student background, engagement, and familiarity with digital task demands. Methods Data were drawn from the linked PIRLS 2016 and ePIRLS 2016 assessments, administered to the same students on consecutive days in 16 participating countries and benchmarking entities. Survey-weighted hierarchical linear regression models were estimated using five plausible values for PIRLS informational reading and ePIRLS achievement. Models were specified sequentially to examine the additional contributions of student background characteristics, digital self-efficacy, and behavioral engagement indicators derived from ePIRLS process data, including pages visited, ad clicks, and task enjoyment. This stepwise approach was used to evaluate whether digital reading performance reflected primarily general reading ability or additional variance associated with digital engagement and familiarity. Results Paper-based reading ability accounted for the majority of variance in ePIRLS digital reading performance, explaining 60.4% of score variance in the pooled analysis. Student background factors contributed a small but significant increment in explained variance, while digital self-efficacy and behavioral engagement indicators added further modest improvements in model fit. Performance was positively associated with pages visited and task enjoyment and negatively associated with off-task ad clicks. While country-level variation existed, paper-based reading remained the dominant predictor across all systems, accounting for 46% to 72% of the variance. Conclusions The findings indicate that ePIRLS primarily reflects traditional reading literacy assessed in a digital environment rather than a substantively distinct digital-reading construct. Process data provided useful supplementary evidence by clarifying how engagement-related behaviors shaped performance. These results have implications for construct validation and for the interpretation of scores from large-scale digital reading assessments.
Abstract This study explores the use of Small Area Estimation (SAE) methods to expand reporting of state-level subgroup performance on National Assessment of Educational Progress (NAEP) assessments without requiring the assessment program to draw larger samples of students and expand test administration. To evaluate the SAE methods, small area estimates of mean subgroup achievement formed from samples too small to meet NAEP’s current reporting requirements are compared with design-based estimates of mean subgroup achievement that satisfy the program’s minimum sample size requirement for reporting. Results demonstrate that several SAE methods yield estimates of mean achievement that are more accurate and reliable than estimates of mean achievement that satisfy NAEP’s current reporting requirement. Policy implications and future directions for research are discussed.
Abstract Background TIMSS 2023 introduced three student-report scales—Digital Self-Efficacy, Environmental Attitudes, and Environmental Behaviors—to assess competencies relevant to 21st-century learning. Valid cross-national comparisons require that these scales function equivalently across education systems. This study evaluated their psychometric properties and measurement invariance to assess their suitability for cross-national use. Methods Data from Grade 8 students across 44 education systems were analyzed. Multiple-group confirmatory factor analysis (MGCFA) tested configural, metric, and scalar invariance, followed by alignment optimization to evaluate approximate invariance and support latent mean comparisons. Models were estimated using robust maximum likelihood with full information maximum likelihood for missing data. Results Configural invariance was supported for all scales, but evidence for metric invariance varied across scales and scalar invariance was not achieved. Alignment optimization indicated strong approximate invariance for Digital Self-Efficacy (10.6% non-invariance), acceptable invariance for Environmental Behaviors (21.7%), and more limited invariance for Environmental Attitudes (28.4%). High R² values (≥ 0.957) suggested that most variation reflected substantive cross-national differences rather than measurement bias. Latent mean comparisons revealed distinct patterns, with Digital Self-Efficacy generally higher in several European and East Asian systems, while environmental measures showed relatively higher levels in some Central Asian and Middle Eastern systems. Conclusions The findings provide initial cross-national support for the TIMSS 2023 scales and highlight the value of alignment optimization for large-scale assessments. Digital Self-Efficacy showed relatively strong comparability, whereas environmental measures were more context-sensitive. Results support cautious interpretation of cross-system differences and point to the need for continued scale refinement and external validation.
Abstract Background Brazil’s participation in PISA 2022 offers a unique opportunity to explore how learning outcomes vary across regions within a decentralized education system. Objective We evaluate whether PISA 2022 can generate statistically reliable state-level mathematics estimates in Brazil and identify key student-, school-, and state-level predictors of performance. Methods We analysed mathematics scores from 10,798 fifteen-year-olds across 598 schools in all 26 Brazilian states and the Federal District. To assess concurrent validity, state-level PISA means were regressed on mean scores from the 2021 Sistema de Avaliação da Educação Básica (SAEB). We then applied a three-level hierarchical linear model—students nested within schools, nested within states—adding predictors stepwise. Results State PISA means accounted for 64% of the variance in SAEB scores (R² = 0.64), showing strong concurrent validity with the national grade 9 assessment. At the student level, mathematics preference (β = 0.44), anticipated effort under grading (β = 0.18), and socioeconomic status (β = 0.14) were the strongest positive predictors, while grade repetition had the largest negative coefficient (β = − 0.43). Private-school attendance was associated with higher scores, corresponding to an estimated difference of approximately four years of schooling. At the state level, after controlling for variables at all levels, only the municipal Human Development Index (HDIm; β = 0.13) remained statistically significant. Conclusions PISA 2022 illuminates sub-national indicators of Brazilian educational quality and uncovers globally recognized, multilayered predictors of mathematics performance. Routinely integrating ILSAs with national assessments can help inform equity-oriented education policies.
Understanding students’ environmental values and behaviors is important for fostering sustainable practices through education. In this study, we investigated the measurement properties and structural relationships of the Students Value Environmental Preservation and Environmentally Responsible Behaviors scales from Trends in International Mathematics and Science Study (TIMSS) 2023 within the Norwegian context, using data from students in Grades 5 and 9. Confirmatory factor analysis (CFA) supported the hypothesized factor structure of both scales, indicating good reliability and validity. Multigroup CFA established partial measurement invariance across genders and showed that girls score significantly higher on environmental values and environmentally responsible behaviors than boys, with the gender gap more pronounced in Grade 9. In a series of structural equation models, we demonstrated that environmental values are a strong, consistent predictor of environmentally responsible behaviors across grades and gender. In contrast, science achievement showed weak or negative associations with responsible behaviors, particularly for Grade 9 boys, suggesting that academic success in science does not necessarily translate into pro-environmental behaviors. These findings highlight the importance of cultivating environmental values as a foundation for responsible behaviors and underscore the need for gender-responsive science education that connects values with real-world applications.
Abstract Background Academic optimism (i.e., academic emphasis, collective efficacy, and faculty trust) describes the striving for a high-achieving learning environment. It has been identified as an important factor for students’ academic achievement at the school level to directly promote the academic achievement of all students, independent of their background. So far, mostly empirical studies using cross-sectional designs have revealed a statistically significant positive relationship between academic optimism and academic achievement. Current research using longitudinal designs at the school level over more than two years and examining elementary schools, a critical stage in a child’s school career, at the same time, is comparatively scarce. This study investigated the association between academic optimism and students’ reading achievement at the school level over five years. Specifically, we examined the school’s academic optimism in 2016 and its relationship to the reading achievement of the fourth-grade students in 2021, and its association with the change in reading achievement from 2016 to 2021. Methods In this study, we used data from school principals and fourth-grade students from N = 111 schools participating in a school panel study in Germany in 2016 and 2021. We focused on analyzing the long-term relationship of all three facets of academic optimism on students’ reading achievement by using linear mixed-effect models and considering central control variables (e.g., socioeconomic status and migration background). Results The results indicated a statistically significant association between collective efficacy in 2016 and students’ reading achievement in 2021, when considering control variables. Higher collective efficacy of schools was associated with higher reading achievement of fourth-grade students five years later. For academic emphasis and faculty trust, no significant relationship with students’ future reading achievement was revealed. Additionally, the reading achievement from 2016 to 2021 was not related to the facets of academic optimism. Conclusion The study reveals important findings regarding the relevance of academic optimism and, particularly, collective efficacy for students’ academic achievement in elementary schools. Additionally, the panel data structure also enabled the detection of the missing protective function of academic optimism against the negative development of achievement, which requires further exploration. Implications for research and practice are discussed.
Abstract Research on how socioeconomic status (SES) and gender interact to affect student achievement has produced contradictory findings. Some studies suggest boys are more vulnerable to socioeconomic disadvantage; others find the opposite. This paper argues that these conflicting results stem from a methodological artifact: because boys report their parents’ education less accurately than girls do, the source of parental education data in large-scale assessments fundamentally shapes estimates of the SES–gender interaction. PISA 2006 and 2009 provide data on 151,269 fifteen-year-old students from 20 countries for whom both parent-reported and student-reported parental education were available. This within-subject design allowed a direct comparison of the parental-education–gender interaction in mathematics achievement, isolating the effect of the data source. The direction of the interaction systematically reverses depending on the data source. When using reliable parent-reported data, the parental-education gradient is steeper for boys in the majority of country-waves (18 of 31), consistent with the “vulnerable boys” hypothesis. In 12 of those 18 cases, the results reverse to indicate “vulnerable girls” when using the more commonly available student-reported proxy data. A sign test confirmed that the interaction estimate was more negative with student data in 28 of 31 country-waves (p < 0.001). The choice of data source is not a minor technical detail but a factor that can reverse conclusions about educational equity.
Abstract Background International large-scale assessments such as TIMSS offer valuable insights into teaching effectiveness. However, student-reported measures often exhibit perceptual variability, raising concerns about their reliability and validity. Methods This study applies a climate strength model to assess not only the average level of teaching quality but also instructional climate strength, operationalised as within-classroom dispersion in student perceptions. Using TIMSS 2023 data from participating educational systems, this study examines two key dimensions of teaching quality, instructional clarity and classroom management, in both mathematics and science, with analytic samples varying by subject and measure availability. Results Multilevel modelling reveals that higher classroom-average teaching quality is associated with better student achievement. Crucially, stronger instructional climate strength further is associated with learning outcomes in some educational systems. In a few educational systems, inconsistent student perceptions appear to diminish the benefits of high-quality teaching, supporting the instructional climate strength hypothesis. Nonetheless, cross-national differences indicate that cultural and pedagogical contexts shape these patterns. Conclusions This study advances methodological approaches for evaluating teaching quality in large-scale assessments, highlighting the importance of considering both the level and consistency of instructional climate. The findings carry important implications for education policy, teacher training, and assessment design, encouraging strategies that promote not only effective but also equitably experienced teaching quality.
Abstract Background Academic resilience, defined as the capacity of socio-economically disadvantaged students to achieve at high academic levels, is a key indicator in international education policy. Large-scale assessments like the Programme for International Student Assessment (PISA) routinely report the prevalence of academic resilience as a summary measure of system quality, combining efficiency (high achievement) and equity (reduced socio-economic disparities). Building on this approach, comparative research has sought to identify school factors that promote resilience, treating it as a property of schools and systems rather than an individual trait. However, most existing evidence relies on cross-sectional data and contemporaneous measures of school characteristics, which may be confounded by other factors, particularly students’ prior achievement. Methods We investigate whether academic resilience in Italy reflects the characteristics of the upper-secondary school that students attend at age 15 or earlier achievement differences. We use unique longitudinal data linking Italian students’ achievement in PISA 2018 (grade 10) to their standardized test scores from grade 8 and 5 (INVALSI, grades 5 and 8; N = 5,323). Since grade 8 marks the end of lower secondary education in Italy, just before students are tracked into different upper secondary schools, controlling for grade 8 achievement allows us to isolate the contribution of the two years of upper secondary schooling captured in PISA. Following OECD reporting, academic resilience is defined as being in the lowest quartile of socio-economic status and the top quartile of mathematics achievement in grade 10. We describe achievement trajectories by socio-economic group and use multivariate models to assess the relative contribution of key school factors identified in PISA (e.g., disciplinary climate or truancy) and prior achievement to academic resilience. Results Socio-economic disparities in mathematics achievement emerge early in the Italian education system. Grade 8 achievement is the strongest predictor of the likelihood that disadvantaged students will be academically resilient in grade 10. Most school-level factors at the upper-secondary level that are measured in PISA show weak or non-significant independent associations once prior achievement is accounted for. Academic success in grade 8 is more predictive of later success for socio-economically advantaged students. Initially high-achieving but less socio-economically advantaged students are more likely to fall behind in their educational careers. Conclusions Academic resilience in upper-secondary schools in Italy appears to reflect achievement differences that are established early and evolve over students’ academic careers. The cross-sectional indicators of resilient students or resilient schools measured in PISA at the upper-secondary level should therefore be best viewed as descriptive indicators rather than measures of school effectiveness.
Abstract Background This study investigates the factors associated with eighth-grade mathematics achievement in Ghana, addressing the research gap by comparing student- and school-level influences. Given the persistently low mathematics performance in Ghana and sub-Saharan Africa, understanding these factors is critical for improving educational outcomes in the region. Methods Using data from the 2011 Trends in International Mathematics and Science Study (TIMSS), a two-level hierarchical linear model was used to analyse the mathematics achievement of 7323 students nested within 161 schools. Student-level variables included confidence, enjoyment, the value of mathematics, and educational expectations. School-level variables included discipline and safety, orderly environments, and emphasis on academic success. Results The analysis revealed that 40.71% of the variance in mathematics achievement was between schools, while 59.29% was attributable to individual students. At the student level, enjoyment of mathematics (β = 24.30, p < .001) was the strongest positive predictor. At the school level, principals’ perceptions of discipline and safety (β = 31.15, p < .001) were the most significant positive predictors of achievement. Conclusion Mathematics achievement in Ghana is associated with both students’ attitudes and the school environment. Enhancing students’ enjoyment and confidence in mathematics, alongside fostering disciplined, safe, and academically focused schools, is essential for improving students’ performance.
Abstract Background Absence of differential item functioning (DIF) is an important piece of evidence to support inferences based on group comparisons of test results. We illustrate how to tailor the DIF identification process following the guiding questions proposed by Sireci and Rios for large scale assessment’s (LSA) specific characteristics by examining DIF between two test forms of the Mathematics test of a Colombian LSA (SABER 11). Methods We investigate the performance of the non-compensatory DIF (NCDIF) index and the Mantel–Haenszel (MH) DIF procedure under large sample sizes and sample size ratios (up to 1:25), and the performance of effect size guidelines under these conditions. These simulations were needed to adequately address the guiding questions for DIF analyses of SABER 11. DIF analyses of SABER 11 test forms were conducted in light of the results of these simulations. Results Type I error is affected, for both procedures, by both the sample size and sample size ratio, as well as by the magnitude of impact between the groups. The joint use of the effect size guidelines helps mitigate this issue without much loss of power given the large sample sizes involved. The DIF analyses of the Mathematics test forms of SABER 11 provide robust evidence that the inferences derived from score comparisons are fair. Conclusions Beyond the immediate implications for the use of SABER 11 tests, the presented case study may help guide practitioners in the assessment of DIF by illustrating how to perform several of the steps involved. Moreover, the simulation studies shed new insights into the frequentist behavior of the two DIF indices under conditions that had not been previously explored but which are applicable to many LSA. Additionally, the results indicate that simulation studies examining the performance of NCDIF, MH, and possibly any DIF statistic, should implement realistic item parameter pools and not only sanitized well-distributed sets of item parameters.
Mathematical proficiency in adolescence is crucial for both individual success and national economic development; yet few studies have examined cross-national differences in how achievement-related beliefs, motivations, and institutional factors affect mathematical outcomes. Specifically, this study filled the gap by investigating the impact of growth mindset, intrinsic motivation, and school autonomy on students’ mathematics performance across five top-performing Asian (Singapore, Macao, Hong Kong, Taipei, and Korea) and five top-performing Western (Switzerland, Ireland, Denmark, the United Kingdom, and Poland) education systems. We used the Programme for International Student Assessment 2022 dataset (N = 66,789) and multilevel mediation analyses to reveal that (a) growth mindset was positively associated with mathematics performance in all five Western economies and two Asian economies (Singapore and Taipei). (b) Intrinsic motivation mediated the pathway from growth mindset to mathematics performance in four Western economies (Ireland, Denmark, the United Kingdom, and Poland) and all five Asian economies. (c) School autonomy exhibited context-dependent moderating effects, strengthening the influence of growth mindset in Korea while amplifying the association between intrinsic motivation and mathematics performance in Singapore. This study highlights the importance of aligning educational interventions that target students’ motivational beliefs with the cultural and institutional contexts in which they are implemented.
