Autonomous vehicles promise safer roads and smoother mobility, yet one stubborn obstacle keeps them from widespread adoption: people do not fully trust them. As automation shifts the human from active driver to supervisory passenger, a new communication problem emerges. The passenger must understand what the vehicle intends to do, while the vehicle must understand what the human is experiencing, whether that is attention, fatigue, distraction, or disengagement. A research team at Tecnologico de Monterrey in Monterrey, Mexico, has now built and tested an integrated framework that tackles both sides of this exchange at once, combining a transparent real-time vehicle interface with a camera-based system that reads a driver’s behavioral state from facial landmarks alone.
The testbed was no simulation. The team equipped a Polaris GEM e6 electric shuttle, a neighborhood electric vehicle upgraded by AutonomousStuff with full drive-by-wire control via PACMod, enabling software-level command of steering, braking, acceleration, and gear shifting. Dual Velodyne VLP-16 LiDARs, forward-facing Triton cameras, and a NovAtel GNSS-IMU system provided perception and localization, all orchestrated by the Autoware.ai software stack on an onboard industrial computer running Ubuntu and a ROS-based modular architecture. On top of this platform, the researchers built a Human-Machine Interface in Foxglove Studio that visualized vehicle speed, driving mode, localization, obstacle detections, and system status in real time, so that a supervisory user could see exactly what the autonomous shuttle perceived and planned.
Latency measurements showed the interface was genuinely real-time. Camera, GNSS, and drive-by-wire feedback topics, including steering, velocity, braking, and gear state, exhibited a mean latency of 5 milliseconds between sensor capture and data availability on the ROS network. The LiDAR perception pipeline, which transforms raw point clouds through ground segmentation, Euclidean clustering, shape estimation, and multi-object tracking, achieved a median end-to-end latency of approximately 78 milliseconds from acquisition to object tracking output. Path-tracking evaluation on a closed campus parking area configured as a small oval track showed a mean absolute lateral error of 0.97 meters, with an RMS of 1.06 meters, confirming that the shuttle could navigate autonomously while streaming its internal state to the user without formal user studies, which were outside the scope of this validation phase.
The second half of the framework, the driver monitoring system, relied on an Intel RealSense L515 depth camera mounted on the rear-view mirror. Processing ran on the vehicle’s onboard computer in Python 3.8, using OpenCV for video acquisition and MediaPipe Face Mesh for landmark extraction. The system delivered 468 three-dimensional facial landmarks per frame at approximately 16.66 frames per second and a resolution of 960 by 540 pixels, from which three interpretable indicators were computed: the Eye Aspect Ratio, or EAR, for eye closure detection; the Mouth Aspect Ratio, or MAR, for yawning detection; and head pose in yaw and pitch, estimated by mapping 2D landmarks such as the nose tip, outer eye corners, mouth corners, and chin onto a canonical 3D facial model through OpenCV’s Perspective-n-Point algorithm. Face detection succeeded in 98.5% of frames across the recording sessions, demonstrating reliable landmark tracking throughout.
A crucial design choice was individual calibration. Before any classification, the participant performed controlled facial actions to establish personalized baselines: maintaining a neutral gaze, opening the mouth in a relaxed state, and orienting the head toward predefined corners of the windshield to record the yaw and pitch angles that define a safe zone for gaze direction. The system stored these averages as thresholds tied to the user’s unique identifier. A structured training protocol then guided the participant through five behavioral states over timed intervals separated by short pauses: neutral, eyes closed for 15 seconds, looking outside the safe zone, yawning, and repeated blinking. A threshold-based finite-state model identified the active state in real time, for example flagging closed eyes when EAR fell below 50% of the calibrated value for more than 3 seconds, a duration consistent with drowsiness detection criteria, and flagging yawning when MAR exceeded 130% of the calibrated baseline.
This protocol generated two datasets: 2,332 processed frames collected indoors in the controlled cabin environment for training, and 2,257 frames collected outdoors, where sun, noise, and wind varied, as a hold-out test set simulating realistic deployment. Statistical analysis confirmed the features were state-specific. Kruskal-Wallis tests with Bonferroni-corrected post hoc comparisons revealed strong significance for closed eyes on EAR, yawning on MAR, and fast blinking on blink rate, with fast blinking defined as more than 30 blinks per minute with each blink lasting more than 0.4 seconds, thresholds grounded in published blink duration and rate ranges. Only the out-of-safe-zone state, measured by head rotation, failed to separate significantly from the neutral condition, foreshadowing the classification challenges that followed.
The team trained classical and deep learning classifiers on the facial features. A linear Support Vector Machine and a Random Forest with 100 estimators and a maximum depth of 3 were trained in scikit-learn with stratified 10-fold cross-validation, while static Artificial Neural Networks and sequential Convolutional and Recurrent Neural Networks, including GRU and LSTM cells, were trained in TensorFlow on an RTX 3060 GPU across a systematic hyperparameter sweep of 1,260 models varying neurons, hidden layers, sequence lengths, and random seeds. Under cross-validation, SVM achieved 91% accuracy and 91% F1-score and RF reached 92% accuracy and 91% F1-score, both comfortably outperforming the threshold-based heuristic model, which yielded 71% accuracy and 61% F1-score on the unshuffled training segments. ROC analysis through the SVM’s One-vs-All configuration showed the strongest discrimination for closed eyes and fast blinking at 98% AUC, followed by yawning at 95%, with out-of-safe-zone at 88% and neutral at 89%.
Generalization told a sharper story. On the unseen outdoor data, SVM held its performance at 90% accuracy and 89% F1-score, exceeding the heuristic baseline of 81% F1-score, with excellent per-state scores for neutral at 95%, closed eyes at 96%, yawning at 93%, and fast blinking at 94%. The main confusion was out-of-safe-zone at 66%, misclassified as neutral in 31% of cases, likely because transitional head rotations during gaze shifts overlapped the calibrated safe-zone limits. The Random Forest, by contrast, dropped to 82% accuracy and 70% F1-score outdoors, performing worse than the threshold-based model, and struggled heavily with closed eyes at 47%, a failure the authors attribute to how the forest’s majority voting defaults to the dominant class when trees encounter ambiguous, overlapping instances, combined with sensitivity of maximum depth to overfitting. Deep learning architectures, tuned to 8 neurons and a sequence size of 5, reached test F1-scores of about 80%, with GRU at 80% and 1D-CNN at 80%, comparable to the heuristic baseline but never surpassing SVM. Interestingly, classical models primarily misclassified out-of-safe-zone instances as neutral, while deep architectures shifted predictions the opposite way, mapping overlapping regions into separable cluster representations that favored minority classes, though a static ANN heavily biased toward non-neutral predictions, managing only 49% accuracy on neutral.
Inference efficiency reinforced the case for interpretability. GPU-accelerated deep learning required 55 milliseconds per prediction, compared with 5 milliseconds for Random Forest and 0.2 milliseconds for SVM on CPU, and all models met the 60-millisecond real-time window. The authors conclude that for multi-class behavioral classification derived from low-dimensional facial metrics, the linear SVM is preferred: its marginalized decision boundary aligns with the structured feature space, offering easier implementation, faster inference, better interpretability, and higher trust than black-box alternatives. The study’s limitations are significant, notably the sample size of one participant in a stationary vehicle, which prevents testing generalization across diverse facial morphologies, ages, and dynamic driving conditions. Still, the personalized calibration protocol successfully reduced cross-session degradation, and the framework demonstrates the feasibility of behaviorally responsive, transparent human-autonomous vehicle interaction, where the machine shows its intent and the human’s state is read through explicit, understandable facial cues, a foundation the team says could support multi-modal feedback, real-time driver alerting, and multi-user calibration in future shared mobility deployments.
Subject of Research: Interpretable driver behavioral state monitoring using facial landmarks for human-autonomous vehicle interaction
Article Title: Interpretable driver behavioral state monitoring using facial landmarks for human-autonomous vehicle interaction
Article References: Candela, M. O., Mandujano-Granillo, J. A., Vidal-Lopez, I., Ramírez-Moreno, M. A., Tudon-Martinez, J. C., & Lozoya-Santos, J. D.-J. (2026). Interpretable driver behavioral state monitoring using facial landmarks for human-autonomous vehicle interaction. Frontiers in Robotics and AI, 13, Article 1896859. https://doi.org/10.3389/frobt.2026.1896859
Image Credits: AI Generated
DOI: 10.3389/frobt.2026.1896859
Keywords: autonomous vehicles, driver monitoring system, facial landmarks, machine learning, human-robot interaction, support vector machines, Interpretable, driver, behavioral, state, monitoring, facial
