Using large language models for temporal relation extraction from pediatric clinical reports.

OBJECTIVESTo evaluate large language models (LLMs) for extracting temporal relations from pediatric rare disease clinical reports to enable automated patient timeline creation.MATERIALS AND METHODSWe developed a temporal relation extraction framework for electronic health records, using 25 clinical reports from a pediatric rare disease hospital. We implemented few-shot prompting with 3 different LLMs in secure environments.RESULTSOur findings reveal that binary classification significantly outperforms multi-class approaches for temporal relation extraction, with best F1 scores reaching 0.70 for simpler relations while more complex relations remain challenging (F1: 0.03-0.40). Mistral 22B emerged as the strongest overall performer, though model superiority varied by relation type.DISCUSSIONThe dramatic performance improvement from reducing cognitive load (binary vs multi-class classification) demonstrates that task formulation critically impacts LLM effectiveness in specialized clinical domains. Our few-shot approach successfully enables temporal relation extraction from French pediatric texts while maintaining data privacy through local deployment, offering a viable methodology for healthcare institutions with strict data governance requirements.CONCLUSIONOur few-shot prompting approach demonstrated promising results in secure environments. This methodology allows technique sharing without exposing sensitive data, advancing research possibilities for clinical natural language processing in restricted settings.KEYWORDSlarge language model, patient timeline, rare diseases, temporal relation extraction© The Author(s) 2025. Published by Oxford University Press on behalf of the American Medical Informatics Association.