Building a good mathematics test is a surprisingly hard optimization problem. Every item in a test bank carries its own content domain, difficulty level, cognitive demand, and point value, and a well-constructed exam must balance all of these simultaneously while staying within a fixed length. A new open-access study in Discover Artificial Intelligence shows that a genetic algorithm — a search method borrowed from evolutionary computation — can do this balancing act automatically, assembling a 25-item mathematical literacy test from a 120-item bank and then validating the result with real student responses. The work, led by Fei Wang of Dalian University of Science and Technology together with Si Wu and Yonghui Zhang, offers a transparent, computationally accessible workflow for schools and institutions that lack the enormous calibrated item banks and psychometric infrastructure of international assessments like PISA.
The design of the test itself is inspired by the PISA 2022 mathematics framework, which defines mathematical literacy not as the reproduction of isolated procedures but as the capacity to use mathematical knowledge and reasoning to solve problems in meaningful personal, occupational, societal, and scientific contexts. The framework emphasizes mathematical reasoning alongside three problem-solving processes — formulating, employing, and interpreting mathematics in real-world situations. The researchers translated this competency-oriented view into an explicit test blueprint with four content domains: Number and Algebra, Geometry, Statistics and Probability, and Comprehensive Practice, each with a target share of the total score points. Importantly, the team did not attempt to reproduce PISA’s official psychometric machinery, which relies on Item Response Theory and large-scale booklet designs; instead, they kept the statistical constraints within Classical Test Theory, where item difficulty is simply the proportion of test-takers who answer an item correctly.
The genetic algorithm works by encoding each candidate test form as a binary chromosome — a string with one position per item in the bank, where a 1 means the item is included and a 0 means it is excluded. The researchers initialized a population of 500 candidate forms, each constrained to contain exactly 25 items, providing a diverse starting pool of item combinations. Every candidate was then scored by a fitness function combining two weighted deviations: the average absolute difference between target and achieved score-point percentages across the four content domains, weighted at alpha = 0.4, and the absolute difference between the target overall difficulty of 0.65 and the difficulty achieved by the candidate form, weighted at beta = 0.6. The weights were chosen from preliminary pilot runs, with greater emphasis placed on matching the statistical difficulty target. A transformation converts the combined deviation into a maximization-oriented fitness value, so candidates with smaller total deviations receive higher scores.
From there, the algorithm runs its evolutionary loop. Parent forms are sampled according to fitness, with selection probabilities derived from the cumulative fitness distribution and duplicate selections avoided when constructing parent pairs. Selected parents undergo crossover and mutation to generate offspring, which are evaluated with the same fitness function. An elitist strategy retains the highest-fitness forms between generations so that high-quality solutions are never lost. The search terminates when the predefined iteration limit is reached or when a feasible high-fitness form satisfies the principal test-length, content, and difficulty constraints. Minimum representation of the required cognitive-process categories is imposed as a hard feasibility constraint, meaning no candidate can survive if it fails to cover the intended problem-solving processes. The item bank itself drew on released PISA-related assessment materials and publicly available educational repositories, with roughly 40 percent easy items, 40 percent medium, and 20 percent difficult, each tagged with content domain, cognitive-process category, preset difficulty, and point value.
The assembled form was then administered to 42 high-school students, sampled through a stratified random procedure based on academic performance bands of high, medium, and low. Multiple-choice items were scored dichotomously, while constructed-response items could earn partial credit according to scoring rubrics. The empirical results were strikingly close to the algorithmic targets. The point-weighted empirical difficulty of the complete form came in at 0.639 against the target value of 0.650 — an absolute deviation of just 0.011. At the item level, 18 of the 25 items, or 72 percent, fell within the expected performance range for their preset difficulty categories, while seven items showed noticeable deviation, a pattern the authors attribute to sampling variability, imperfect CTT difficulty estimates, or interactions between item content and the particular validation group.
Correlation analysis provided convergent quantitative evidence that the intended difficulty structure was preserved. Empirical scoring rates were strongly and negatively associated with ordinal difficulty codes, where easy = 1, medium = 2, and difficult = 3, yielding r = −0.823 with p < 0.001 — the expected negative sign because larger codes denote more difficult items and therefore lower scoring rates. When the same scoring rates were correlated with the preset expected scoring-rate values used in the item bank, the association was equally strong and positive, r = 0.827 with p < 0.001. These two statistics describe the same monotonic relationship from opposite coding directions, and together they demonstrate that the algorithm’s difficulty assignments tracked the observed performance of real students remarkably well.
Content coverage was less exact than difficulty matching, which is consistent with the stated weighting priorities. The target content percentages for the four domains were 44.00 percent for Number and Algebra, 33.00 percent for Geometry, 8.00 percent for Statistics and Probability, and 15.00 percent for Comprehensive Practice. The assembled form achieved 40.00, 41.34, 9.33, and 9.33 percent respectively, giving a mean absolute content-domain deviation of 4.835 percentage points. The largest discrepancy occurred in Geometry, which exceeded its target by 8.34 percentage points, while Comprehensive Practice fell 5.67 points below target. Because beta = 0.6 assigned more weight to difficulty than alpha = 0.4 assigned to content balance, this pattern aligns with the objective priorities, though the authors caution that it should not be read as causal proof of the weighting effect without repeated sensitivity experiments.
The validation sample itself performed strongly on the assembled form. The mean score was 94.157 with a median of 94.987 and a standard deviation of 11.459, over a maximum total score of 150 points. Skewness was 0.464, indicating only modest positive asymmetry, while kurtosis of 0.052 was close to a mesokurtic distribution. Five students, or 11.9 percent, were classified at proficiency Level 3 and 37 students, or 88.1 percent, at Level 4, with no students at Levels 1, 2, or 5. This concentration at Level 4 indicates that the validation group performed relatively well on this particular form, but it also limits how representative the results are of a broader population and reduces precision at the lower end of the ability distribution.
The authors are careful to frame the contribution as an application-level implementation and validation of a constrained automated test assembly workflow, not a claim that genetic algorithms themselves are novel. Prior research has shown that mixed-integer programming can provide exact or near-exact optimization when constraints can be expressed in a tractable mathematical form, while genetic algorithms offer a flexible heuristic search that is relatively easy to adapt when constraints are heterogeneous. The relative advantage of a GA in this context is implementation flexibility; its limitation is that heuristic search does not guarantee the global optimum and may be sensitive to the fitness specification and stopping criteria. For classroom and institutional settings where stable IRT calibration is unavailable, a CTT-based GA provides a transparent and computationally accessible alternative.
Several limitations constrain generalization. Empirical validation rested on 42 valid response records from a limited regional sample, so both item difficulty estimates and proficiency distributions may be sample-dependent. The assembly procedure uses CTT difficulty information rather than IRT parameters, limiting invariance across populations and preventing direct comparison with official PISA scaling. The 120-item bank is moderate in size and may not provide enough alternative items to satisfy all constraints simultaneously, and the fitness weights were selected heuristically without a complete repeated-run sensitivity dataset. Future work should validate the method with larger and more diverse samples, expand and independently calibrate the item bank, incorporate IRT parameters and test information constraints, compare GA performance with mixed-integer programming under repeated runs, and examine multiple-form parallelism, item exposure, subgroup fairness, and external validity. Until then, the method should serve as a decision-support tool that reduces manual search effort — with educators still reviewing the resulting forms for curriculum alignment, accessibility, and fairness before administration — rather than a replacement for expert judgment or psychometric calibration. As evidence of feasibility, the study makes a compelling case that evolutionary search can bring PISA-style competency assessment within reach of ordinary classrooms.
Subject of Research: Genetic algorithm based automated test assembly for mathematical literacy assessment inspired by the PISA framework
Article Title: Genetic algorithm based automated test assembly for mathematical literacy assessment inspired by the PISA framework
Article References: Wang, F., Wu, S., & Zhang, Y. (2026). Genetic algorithm based automated test assembly for mathematical literacy assessment inspired by the PISA framework. Discover Artificial Intelligence, 6(1), Article 1529. https://doi.org/10.1007/s44163-026-02161-8
Image Credits: AI Generated
DOI: 10.1007/s44163-026-02161-8
Keywords: Genetic, algorithm, based, automated, test, assembly, mathematical, literacy, assessment, inspired, PISA, framework
