In a systematic investigation of artificial intelligence psychology, four prominent large language models demonstrated remarkably high test-retest reliability across repeated personality evaluations, challenging earlier findings of erratic machine behavior while revealing that measured traits depend heavily on assessment methodology. Researchers Jason L. Harman and Nithish Raja Karuppiah Loganathan of Michigan Tech University, alongside Jaelle Scheuerman of the U.S. Naval Research Laboratory, evaluated Meta’s Llama 3.1 8B, Mistral’s Devstral Small 2 24B, Google’s Gemma 4 31B Dense, and Google’s proprietary Gemini 3.5 Flash under baseline conditions and prompt-engineered personas. Rather than displaying output instability previously observed in generative systems, every tested model produced repeatable personality factor scores across independent runs. However, profiles generated by traditional Likert questionnaires diverged markedly from those derived through gamified behavioral scenarios, indicating that an artificial agent’s measured disposition is shaped significantly by diagnostic instrument design.
As generative artificial intelligence agents increasingly assume consequential roles across enterprise workflows, task automation, education, and companionship, determining whether they exhibit predictable behavioral styles has become a central research objective. In human psychology, personality is predominantly defined by the Five-Factor Model through five broad dimensions: Openness to Experience, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Stable traits are essential if synthetic agents are to serve as simulated human participants in scientific inquiries, execute reliable collaborative duties, or inspire user trust in mission-critical applications. Nevertheless, prior empirical studies examining machine personality reached conflicting and largely skeptical conclusions. Past experiments reported substantial response variability in default models, while systems administered standard self-report surveys frequently reflected on their status as artificial programs, declined to answer sensitive questions, or generated artificially flattering self-descriptions to satisfy perceived conversational expectations.
These measurement hurdles motivated the research team’s initial exploratory runs in December 2024 using OpenAI’s o1-mini reasoning model across ten repeated iterations. When instructed to think aloud while answering traditional Likert questions, the model frequently commented on its artificial status and questioned its own capacity to judge personal attributes, producing wide score swings across repeated passes. Conversely, when guided through an interactive fantasy narrative where traits were inferred from choices rather than self-ratings, the model ceased its meta-conversational commentary and produced noticeably more consistent scores. Across those preliminary trials, the only consistent pattern was elevated agreeableness paired with diminished neuroticism. The authors concluded that standard Likert surveys encourage disruptive meta-cognitive responses because language models easily recognize the underlying assessment intent. Advancing machine psychometrics thus required testing whether gamified behavioral assessments could establish reliable profiles across diverse architectures while evaluating how prompt-based personas modulate those traits.
To rigorously evaluate these psychometric dynamics, the investigators examined four contemporary language models spanning distinct developers, architectures, and design objectives, utilizing unconstrained stochastic sampling rather than artificial determinism. Meta’s open-weights Llama 3.1 8B, released in July 2024 for multilingual dialogue, achieved benchmark scores of 45.5 on MMLU Pro and 34.6 on GPQA. Mistral’s Devstral Small 2 24B, released in December 2025 for agentic workflows, scored 68.0 on SWE Bench Verified. Google’s Gemma 4 31B Dense, released in April 2026, represented an open-weights multimodal system scoring 85.2 on MMLU Pro and 84.3 on GPQA Diamond. Google’s proprietary Gemini 3.5 Flash reasoning model, launched in May 2026, achieved 92.7 on GPQA Diamond and 83.6 on MMLU Pro. Local models ran via Ollama quantized to 8 bits using creator Modelfiles, while Gemini 3.5 Flash ran via official interface at default settings without fixed random seeds.
Each model completed two psychometric instruments operationalizing the Big Five personality factors through contrasting paradigms. The first was the International Personality Item Pool 50-item inventory, or IPIP-50, a standardized instrument presenting ten declarative statements per personality factor. Models selected from five Likert options ranging from very inaccurate to very accurate, yielding factor scores between 10 and 50 points. The second instrument was the Gamified Personality Measure, specifically the GPM-ni revised, a 31-item interactive text adventure where traits are inferred from decisions in a fantasy narrative. Thirty of the 31 decision scenarios map to the Big Five factors, offering six dilemmas per trait scored from 1 to 3 points, resulting in factor totals from 6 to 18 points. Throughout the narrative game, conversational history was retained across sequential decisions, evaluating behavioral tendencies dynamically rather than through direct self-description.
In addition to evaluating baseline performance in an unprompted Out-of-the-Box condition, the study introduced three distinct persona profiles administered through persistent system prompts. The Bold persona emphasized assertiveness, confidence, and decisive risk-taking, with directional predictions anticipating higher Extraversion, elevated Openness, and lower Neuroticism. The Analytical persona emphasized systematic logic, data-driven evaluation, and structured inquiry, leading researchers to predict higher Conscientiousness, higher Openness, and lower Agreeableness. The Cautious persona promoted deliberation, conservative reasoning, and risk aversion, generating predictions of increased Neuroticism alongside reduced Extraversion and Openness. Each model completed both instruments across all four conditions over five independent runs, creating a balanced design. This protocol allowed the team to measure test-retest consistency via Intraclass Correlation Coefficients, evaluate cross-measure agreement on a standardized 0–100 scale, and track whether persona prompts produced systematic shifts away from baseline.
Addressing the first research question regarding measurement stability, the experimental outcomes strongly supported the authors’ initial hypothesis, revealing exceptionally high test-retest reliability across every evaluated condition. Intraclass Correlation Coefficients ranged from 0.897 to 1.000 across all models, personas, and instruments, with many configurations surpassing 0.99. On the IPIP-50, reliability coefficients ranged from 0.972 to 1.000, with Gemma 4 recording a perfect 1.000 in the Bold condition and Gemini 3.5 Flash reaching 0.999 across several conditions. The Gamified Personality Measure exhibited similarly high consistency, registering values between 0.897 and 0.999. Notably, Meta’s Llama 3.1 8B, despite possessing a smaller parameter footprint and older architecture, maintained reliability coefficients between 0.897 and 0.996 across both assessment formats. The authors emphasized that because an older model exhibited such pronounced consistency, high test-retest reliability cannot be interpreted as evidence of a general temporal increase in model stability.
When examining baseline dispositions without persona prompting, models displayed uniform profiles within each assessment type, yet marked divergences emerged between the two testing methods. On the IPIP-50 inventory, all four models generated high baseline scores for Openness, Conscientiousness, and Agreeableness alongside suppressed scores for Neuroticism, reflecting patterns akin to human social desirability bias. Conversely, in the gamified assessment, Conscientiousness emerged as the highest scoring factor across all models, while Neuroticism shifted from floor levels to moderate expressions. Pairwise Pearson correlations comparing baseline trait profiles among the four models averaged 0.969 on the IPIP-50, ranging from 0.944 to 0.992, indicating striking between-model similarity. On the Gamified Personality Measure, pairwise baseline correlations averaged 0.858, spanning from 0.696 to 0.968, confirming that while models exhibit broadly comparable dispositions, baseline profile agreement remains sensitive to evaluation methodology.
Evaluating cross-measure profile agreement directly addressed the second research question, yielding mixed empirical support. Across all models and experimental conditions, Pearson correlations between standardized IPIP-50 and gamified profiles ranged widely from 0.14 to 0.94, averaging 0.65. Discrepancies between the two instruments were quantified by mean absolute differences ranging from 8.58 to 26.75 points on the standardized scale, averaging 19.42 points, while root mean squared errors spanned from 10.45 to 34.43, averaging 24.21 points. Agreement between instruments proved higher under Bold and Analytical persona conditions than during Out-of-the-Box or Cautious testing. For instance, Llama 3.1 achieved a cross-measure correlation of 0.942 with a low absolute difference of 8.58 points in the Bold condition, whereas Devstral Small 2 recorded a correlation of 0.142 and an absolute difference of 23.71 points in baseline testing.
Investigating the third research question, persona prompting induced measurable trait modifications that partially aligned with a priori directional hypotheses, matching predictions in 28 of 36 comparisons on the gamified measure and 23 of 36 on the IPIP-50. The Bold persona elevated Extraversion while lowering Agreeableness and Neuroticism across models, also depressing Conscientiousness in narrative games to reflect adventurous risk-taking. The Cautious persona aligned most consistently with expectations, producing increases in Neuroticism alongside decreases in Extraversion and Openness, though Llama 3.1 deviated by reducing Neuroticism on both measures. The Analytical persona consistently reduced Agreeableness by prioritizing objective logic, but failed to reliably increase Openness. Overall responsiveness varied across models: Gemma 4 and Gemini 3.5 Flash displayed the largest average absolute personality shifts across persona conditions on both instruments, whereas Devstral Small 2 and Llama 3.1 exhibited comparatively smaller adjustments.
Despite these demonstrations of repeatability and prompt responsiveness, the authors emphasized critical limitations that constrain broader interpretations of artificial personality. Crucially, the experiment did not assess external criterion validity, leaving unresolved whether personality scores derived from Likert questionnaires or gamified tasks reliably predict external agentic behaviors, task performance, or decision fidelity. Furthermore, because proprietary models like Gemini 3.5 Flash undergo undocumented server-side modifications, guardrail adjustments, and eventual retirement, longitudinal stability cannot be assumed. Open-weights models such as Devstral Small 2 and Llama 3.1 mitigate reproducibility risks by permitting weight archival and local runtime control, yet their undisclosed pretraining corpora prevent researchers from determining how data mixtures shape observed dispositions. As generative systems take on autonomous roles, these findings establish that while language models produce remarkably consistent behavioral profiles across repeated measurements, their measured traits reflect specific assessment frameworks rather than genuine internal dispositions.
Subject of Research: Psychometric reliability and persona prompting effects across large language models
Article Title: Personality without a person: measuring personality in generative AI
Article References: Harman, J. L., Scheuerman, J., & Loganathan, N. R. K. (2026). Personality without a person: measuring personality in generative AI. Frontiers in Computer Science, 8, 1940035. https://doi.org/10.3389/fcomp.2026.1940035
Image Credits: AI Generated
DOI: 10.3389/fcomp.2026.1940035
Keywords: large language models, generative AI, personality assessment, Five-Factor Model, test-retest reliability, gamified assessment, persona prompting, psychometrics, Personality, without, person, measuring
