A multitask vision-language artificial intelligence framework evaluated across three academic medical centers demonstrated the ability to interpret dynamic abdominal ultrasound cine loops, draft factual diagnostic reports, and predict surgical interventions for gallbladder disease. Reported by researchers from institutions including Johns Hopkins University, Stanford University, and the University of Colorado, the system addresses persistent clinical bottlenecks in right upper quadrant ultrasound imaging. While ultrasound serves as the preferred first-line imaging modality for hepatobiliary conditions such as gallstones, cholecystitis, and bile duct dilation, its real-world diagnostic reliability is constrained by operator-dependent image acquisition, acoustic artifacts, visual noise, and significant interpretation variability among interpreting radiologists.
The study utilized a comprehensive multicenter dataset comprising 11,001 right upper quadrant ultrasound text-image series, reflecting the complex case mix encountered in realistic emergency department environments. The primary development cohort was collected from Johns Hopkins University and encompassed 9,189 ultrasound examinations gathered between July 31, 2016, and December 31, 2023, representing 8,864 unique patients with a median age of 55 years and including 4,977 female and 3,881 male individuals. For independent external validation, the authors curated two distinct cohorts that were not involved in model training: an external dataset of 1,704 cases comprising 90,616 images from the University of Colorado, representing 2,317 unique patients with a median age of 48 years, and a second external dataset of 108 cases comprising 3,240 images from Stanford University. Across all sites, examinations were included based on clinical indications, while studies with severe technical limitations precluding diagnostic interpretation were systematically excluded.
To convert unstructured narrative text into verifiable supervised training targets, the investigators developed an automated language processing pipeline powered by a locally deployed Llama 3 70B Instruct model to parse the narrative findings and impression sections across all 11,001 clinical reports. The automated system extracted and standardized diagnostic findings into 40 predefined medical categories spanning hepatic, biliary, renal, pancreatic, vascular, and peritoneal compartments, identifying 37,849 positive findings with an average of 3.4 positive entities per patient. When evaluated against a manually verified gold-standard subset of 100 ultrasound reports, the automated labeler attained a label-level micro-accuracy of 99.60 percent, a report-level strict accuracy of 83.00 percent, a micro-precision of 96.50 percent, a micro-recall of 98.90 percent, and a micro-F1 score of 97.68 percent, yielding only 17 discordant errors across 4,200 evaluated labels. To maintain robust statistical power and avoid severe category imbalance, the team focused subsequent classification modeling on the 16 clinical conditions that exhibited a prevalence exceeding 5 percent.
Handling dynamic imaging inputs required developing a standardized spatial and temporal preprocessing pipeline tailored to clinical ultrasound cine loops, which naturally varied from fewer than 20 frames to over 200 frames. The engineering pipeline applied uniform temporal sampling to reduce longer recordings into a standardized sequence of 32 evenly spaced frames across the early, intermediate, and late phases of each examination, while applying cyclical temporal padding to shorter sequences. The resulting standardized frames were resized using bicubic interpolation to a uniform resolution of 224 by 224 pixels and had pixel intensity values scaled between 0.0 and 1.0, ensuring that downstream deep learning architectures received temporally consistent, artifact-free representations of abdominal anatomy and dynamic transducer sweeps.
The first functional stage of the agent framework, termed the abnormality classification module, utilized a domain-specific Ultrasound Foundation Model featuring a Vision Transformer architecture with a patch size of 16 by 16 pixels. This visual encoder extracted 768-dimensional feature vectors from each of the 32 standardized frames, which were subsequently processed by an attention-based feature aggregator that dynamically weighted individual frames into a consolidated 768-dimensional patient-level embedding before final linear classification. Training incorporated asymmetric loss functions, AdamW optimization with plateau-based learning rate decay, gradient clipping, and orthogonal gradient surgery to avoid inter-task interference. When evaluated on 917 internal test cases from Johns Hopkins University, the classification module achieved a macro area under the receiver operating characteristic curve of 0.820 and a micro area under the curve of 0.865 across the 16 diagnostic categories. Ablation experiments indicated that this domain-pretrained transformer framework consistently surpassed a standard ImageNet-pretrained ResNet-50 visual feature extractor across representative organ systems.
Detailed disease-level performance metrics demonstrated high discriminative capacity for prevalent urgent pathologies on the internal test set, with right pleural effusion achieving an area under the curve of 0.926, ascites reaching 0.904, and hepatic steatosis or fatty liver reaching 0.904, while 12 additional diagnostic categories maintained values between 0.70 and 0.90. The lowest classification performance occurred for suboptimal visualization of the pancreas, which yielded an area under the curve of 0.670. To emulate high-specificity clinical triage conditions, operating thresholds calibrated on the internal test cohort at fixed 90 percent and 95 percent specificities demonstrated preserved diagnostic sensitivity, yielding 0.814 for pleural effusion, 0.772 for hepatic steatosis, 0.752 for cholelithiasis, and 0.740 for ascites. When these fixed operating cutoffs were deployed without modification to external testing sites, the classifier maintained generalizability, achieving a macro area under the curve of 0.794 across the 1,704 University of Colorado examinations and 0.775 across the 108 Stanford University cases, with correlation analyses confirming that variations in institutional disease prevalence did not significantly diminish model discrimination.
The second stage of the framework focused on automated radiology report generation, translating visual features and disease probabilities into coherent, clinically structured narratives. The investigators instruction-tuned three distinct multimodal architectures using 8,269 internal training studies: an UltrasoundQwen model based on the Qwen2.5-VL 7-billion-parameter architecture, alongside two M3D variants paired with either Llama-3.1 8B or Phi-3-mini-4k language backbones. Fine-tuning utilized Low-Rank Adaptation with rank 16 and alpha 32, exposing models to conversational prompts, multi-frame video inputs, and paired binary visual question-answer checks. Across lexical overlap evaluations on the internal test cohort, UltrasoundQwen significantly outperformed the alternative models across all tested measures, registering a BLEU-1 of 0.5902, BLEU-4 of 0.3940, ROUGE-1 of 0.6572, ROUGE-L of 0.5527, and METEOR of 0.5506. Similar lexical advantages were retained on external validation cohorts, with the model achieving a BERT-F1 score of 0.8798, BLEU-1 of 0.4082, and ROUGE-1 of 0.4564 on the University of Colorado dataset.
Recognizing that surface linguistic overlap does not capture medical validity, the research team instituted advanced clinical fidelity evaluations using the Feature-Oriented Radiology Task Evaluation framework and the automated DocLens claim-verification metric. The feature-oriented assessment measured precision and recall across four defined semantic dimensions, specifically evaluating descriptors of severity degree, anatomical landmarks, primary imaging features, and final diagnostic impressions. On internal testing, UltrasoundQwen attained a DocLens factual verification score of 0.6094 and an average clinical feature F1-score of 0.6539, surpassing competing model variants in factual precision. On the external Stanford dataset, UltrasoundQwen led with a DocLens score of 0.5470, while UltrasoundPhi achieved an average feature score of 0.5292. Across the University of Colorado cohort, UltrasoundQwen again led the cohort with a DocLens score of 0.5904 and an average clinical feature score of 0.5337, confirming its capacity to synthesize accurate anatomical relationships and diagnostic impressions across disparate healthcare settings.
To evaluate downstream clinical decision support, the researchers configured a third module using a Llama 3 model tasked with predicting whether patients required surgical cholecystectomy. The clinical cohort comprised 2,170 patients diagnosed with cholecystitis, partitioned into 1,736 training cases and 434 test cases, where procedural billing codes in electronic records defined true surgical outcomes. When human reports were replaced entirely with machine-generated narratives drafted by the vision-language model, the predictive pipeline achieved an accuracy of 0.718 and a recall of 0.602. Ablation analysis revealed that combining baseline comorbidity codes with narrative reports provided substantial predictive gains over patient demographics alone, demonstrating that machine-generated narratives successfully captured and transferred crucial actionable findings required for surgical triage.
The clinical acceptability and realism of the generated documentation were further scrutinized through blinded reader studies structured as radiologist Turing tests involving 12 participating clinicians. When reviewing unedited artificial intelligence reports alongside authentic human documentation, radiologists distinguished the synthetic drafts with an average accuracy of 0.85, spending an average of 1.18 minutes reviewing each case. However, when evaluating reports that had undergone rapid post-editing by an experienced radiologist, which required an average modification time of 68.4 seconds per case to rectify factual phrasing or minor anatomical omissions, detection accuracy dropped sharply to 0.46, rendering the refined drafts statistically indistinguishable from human-written reports. For mixed evaluations combining unedited and edited cases, average accuracy in discerning authentic human reports was 0.61 using text alone and 0.69 when full ultrasound cine clips were provided, with experienced attending radiologists gaining greater discriminating advantage from image context than clinical trainees.
Despite the strong performance reported across diagnostic classification, reporting, and surgical triage, the authors delineated important methodological constraints that temper immediate clinical translation. The retrospective study design relied on weakly supervised automated text extraction from historical records, which might overlook nuanced diagnostic distinctions captured in free-form prose. Furthermore, the development pipeline prioritized acute emergency biliary and hepatic conditions, intentionally omitting major oncologic findings such as primary liver masses and gallbladder polyps from the current label schema. The cholecystectomy recommendation component was trained on a restricted single-center subset of cholecystitis patients, requiring prospective validation across broader surgical populations. The authors also cautioned that classification operating thresholds derived at Johns Hopkins University represent retrospectively evaluated experimental cutoffs rather than universal deployment thresholds, necessitating rigorous prospective trials to verify performance stability across diverse scanner manufacturers, acoustic protocols, and clinical settings before routine integration into clinical practice.
Subject of Research: Multimodal artificial intelligence for multi-frame right upper quadrant ultrasound interpretation and surgical decision support
Article Title: A multitask framework for automated multi-frame right upper quadrant ultrasound interpretation and clinical decision support
Article References: Guo, H., Li, C.-Y., Wang, Y., Wang, R., Dai, Y., Peng, Q., Cao, D., Zhong, Z., Vu, T., Zhao, L., Zhu, C., Tan, C., Schick, J., Kwak, S., Sedaghat, F., Azadi, J., Facciola, J., Feng, J., Oncel, D., … Bai, H. (2026). A multitask framework for automated multi-frame right upper quadrant ultrasound interpretation and clinical decision support. Nature Communications, 17(1), Article 10587. https://doi.org/10.1038/s41467-026-77498-w
Image Credits: AI Generated
DOI: 10.1038/s41467-026-77498-w
Keywords: Right Upper Quadrant Ultrasound, Vision-Language Model, Artificial Intelligence, Medical Imaging, Emergency Radiology, Ultrasound Foundation Model, Automated Report Generation, Cholecystectomy Prediction, Cholecystitis, Turing Test, Deep Learning, Clinical Decision Support, Hepatobiliary Imaging, Diagnostic Accuracy, DocLens Evaluation, FORTE Framework, Multi-Frame Imaging, Cross-Site Validation
