Precision Health in the Age of LLMs

Medicine today is imprecise. Among the top 20 drugs in the U.S., up to 80% of patients are non-responders. The goal of precision health is to provide the right intervention for the right people at the right time. The key to realize this dream is to develop a data-driven, learning system that can instantly incorporate new health information to optimize care delivery and accelerate biomedical discovery. In reality, however, the health ecosystem is mired in overwhelming unstructured data and excruciating manual processing. For example, in cancer, standard of care often fails, and clinical trials are the last hope. Yet less than 3% of patients can find a matching trial, whereas 40% of trial failures simply stem from insufficient recruitment. Discovery is painfully slow as a new drug may take billions of dollars and over a decade to develop.

In this tutorial, we will explore how large language models (LLMs) can serve as a universal structuring tool to democratize biomedical knowledge work and usher in an intelligence revolution in precision health. We first review background for precision health and give a broad overview of the AI revolution that culminated in the development of large language models, highlighting key technical innovations and prominent trends such as consolidation of AI methods across modalities. We then give an in-depth review of biomedical LLMs and precision health applications, with a particular focus on scaling real-world evidence generation and drug discovery. To conclude, we discuss key technical challenges (e.g., bias, hallucination, cost), societal ramifications (e.g., privacy, regulation), as well as exciting research frontiers such as prompt programming, knowledge distillation, multi-modal learning, causal discovery.

As a resource, we provide a non-exhaustive list of papers and other resources that we referred to during the tutorial. We broadly categorize resources into three categories aligned with the structure of the tutorial: Precision Health, The Intelligence Revolution, LLMs for Precision Health, Application Challenges, and Research Frontiers.

Precision Health

• AI in health and medicine
• Foundation models for generalist medical artificial intelligence

LLMs for Precision Health

GPT-4 in Medicine

• The AI Revolution in Medicine
• Capabilities of GPT-4 on Medical Challenge Problems

Biomedical LLMs

• GPT-4: The AI Revolution in Medicine
• Med PaLM: Large Language Models Encode Clinical Knowledge
• Med PaLM 2: Towards Expert-Level Medical Question Answering with Large Language Models
• BioGPT: BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• PubMedGPT / BioMedLM: Stanford CRFM Introduces PubMedGPT 2.7B
• BioMedGPT: BioMedGPT: Open Multimodal Generative Pre-trained Transformer for Biomedicine
• BioMegatron: BioMegatron: Larger Biomedical Domain Language Model
• GatorTronGPT: A Study of Generative Large Language Model for Medical Research and Healthcare
• Galactica: Galactica: A Large Language Model for Science
• PubMedBERT: Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• ClinicalBERT: Publicly Available Clinical BERT Embeddings
• BioBERT: BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• BioLinkBERT: LinkBERT: Pretraining Language Models with Document Links
• SciBERT: SciBERT: A Pretrained Language Model for Scientific Text
• DoT5: Compositional Zero-Shot Domain Transfer with Text-to-Text Models
• SciFive: SciFive: a text-to-text transformer model for biomedical literature

LLMs for Real-World Evidence

• The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
• Large language model (ChatGPT) as a support tool for breast tumor board
• Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study
• A.I. May Someday Work Medical Miracles. For Now, It Helps Do Paperwork.

LLMs for Drug Discovery

• Drug discovery companies are customizing ChatGPT: here’s how
• AI-based language models powering drug discovery and development

Application Challenges

Bias

• Coding Inequity: Assessing GPT-4’s Potential for Perpetuating Racial and Gender Biases in Healthcare
• ‘Doctors need to get on top of this’: GPT-4 displays bias in medical tasks

Hallucinations

• Survey of Hallucination in Natural Language Generation
• How Language Model Hallucinations Can Snowball
• Mitigating Language Model Hallucination with Interactive Question-Knowledge Alignment
• Evaluating Object Hallucination in Large Vision-Language Models

Research Frontiers

Prompt Programming

• Prompt Engineering
• Language Models are Few-Shot Learners
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• ReAct: Synergizing Reasoning and Acting in Language Models
• Self-Refine: Iterative Refinement with Self-Feedback
• Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
• Prompt Engineering Guide Papers
• ACL 2023 Tutorial on Complex Reasoning in Natural Language
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
• Augmented Language Models: a Survey

Retrieval-Augmented Generation (RAG)

• ACL 2023 Tutorial on Retrieval-based Language Models and Applications
• Augmented Language Models: a Survey
• Foundation models for generalist medical artificial intelligence

Knowledge Distillation

• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Alpaca: A Strong, Replicable Instruction-Following Model
• Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality
• Instruction Tuning with GPT-4
• Textbooks Are All You Need
• Knowledge Distillation of Large Language Models
• The False Promise of Imitating Proprietary LLMs
• LAMPP: Language Models as Probabilistic Priors for Perception and Action
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Multi-modal learning

• Large-Scale Domain-Specific Pretraining for Biomedical Vision-Language Processing
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Multilingual translation for zero-shot biomedical classification using BioTranslator
• MolXPT: Wrapping Molecules with Text for Generative Pre-training
• Multimodal biomedical AI
• Foundation models for generalist medical artificial intelligence
• CVPR 2023 Tutorial on Recent Advances in Vision Foundation Models

Causal Discovery

• A Causal Roadmap for Generating High-Quality Real-World Evidence
• Causal Reasoning and Large Language Models: Opening a New Frontier for Causality
• Can Large Language Models Infer Causation from Correlation?
• Causal-BERT : Language models for causality detection between events expressed in text
• Causal Discovery with Language Models as Imperfect Experts