top of page
  • Writer's pictureHrituraj Singh

Publication: Distilling LLMs for matching patients to clinical trials

Quick Summary

We show parity with proprietary large language models (LLMs) after fine-tuning an open-source LLM on a limited and synthetic dataset.

Publication Date: April 19, 2024



The recent success of large language models (LLMs) has paved the way for their adoption in the high-stakes domain of healthcare. Specifically, the application of LLMs in patient-trial matching, which involves assessing patient eligibility against clinical trial’s nuanced inclusion and exclusion criteria, has shown promise. Recent research has shown that GPT-3.5, a widely recognized LLM developed by OpenAI, can outperform existing methods with minimal ’variable engineering’ by simply comparing clinical trial information against patient summaries. However, there are significant challenges associated with using closed-source proprietary LLMs like GPT-3.5 in practical healthcare applications, such as cost, privacy and reproducibility concerns. To address these issues, this study presents the first systematic examination of the efficacy of both proprietary (GPT-3.5, and GPT-4) and open-source LLMs (LLAMA 7B,13B, and 70B) for the task of patient-trial matching. Employing a multifaceted evaluation framework, we conducted extensive automated and human-centric assessments coupled with a detailed error analysis for each model. To enhance the adaptability of open-source LLMs, we have created a specialized synthetic dataset utilizing GPT-4, enabling effective fine-tuning under constrained data conditions. Our findings reveal that open-source LLMs, when fine-tuned on this limited and synthetic dataset, demonstrate performance parity with their proprietary counterparts. This presents a massive opportunity for their deployment in real-world healthcare applications. To foster further research and applications in this field, we release both the annotated evaluation dataset along with the fine-tuned LLM – Trial-LLAMA – for public use.


Mauro Nievas, Aditya Basu, Yanshan Wang, Hrituraj Singh, Distilling large language models for matching patients to clinical trials, Journal of the American Medical Informatics Association, 2024;, ocae073,


Image by National Cancer Institute

Next Page


601 Montgomery St, Suite 1100
San Francisco, CA 94111


Privacy Policy

All Content Copyright 2024

bottom of page