AMA News Wire
Choosing the right AI model for prognostication, treatment
Aug 13, 2024
Augmented intelligence (AI), often called artificial intelligence, stands to have a profound effect on every aspect of the clinical process, from prevention to prognostication. Physicians recognize this too—an AMA survey of more than 1,000 doctors found nearly two-thirds can see AI’s potential benefits.
A CME series featured on the AMA Ed Hub™ introduces learners to foundational principles in AI and machine learning, a subdomain of AI that enables computers to learn patterns and relationships from data without being explicitly programmed by people. Developed by the AMA ChangeMedEd initiative and the University of Michigan DATA-MD team and geared toward medical students, this series is also suitable for residents, fellows, practicing physicians and other health professionals.
The fourth module in the series, “AI for Prognostication and Treatment,” explores how to interpret the results of a prognosis study of an AI model and apply it to a clinical case. Physicians are accustomed to reviewing articles about the efficacy of drugs or medical devices. Similarly, doctors should also review articles about AI tools to ensure a given tool is appropriate to apply to a given patient or population, and to understand the tool’s limitations or potential risks. Such review requires a similar, yet targeted, skill set as outlined in the module.
The AMA has developed new advocacy principles that build on current AI policy. These new principles (PDF) address the development, deployment and use of health care AI. Meanwhile, AMA Ed Hub also features a 16-credit CME course on artificial and augmented intelligence in health care.
How AI models work
How AI models work
Clinical prediction models can play a pivotal role in prognostication. They work by using statistical techniques to analyze large patient cohorts and then estimate the possibility of specific outcomes based on clinical data.
“These models enhance prognostic accuracy by identifying patterns and relationships between patient characteristics and outcomes,” the module says. “However, these models may be limited by the quality and nature of the data used in their development.”
The Prediction model Risk Of Bias Assessment Tool (PROBAST) is a tool for assessing the quality of prediction models by evaluating the risk of bias in studies that develop or validate those models. Making use of PROBAST guidelines, this CME module explores how to interpret the results of prognosis studies that include Al and apply them to clinical cases.
From AI implementation to EHR adoption and usability, the AMA is making technology work for physicians, ensuring that it is an asset to physicians—not a burden.
Learn more with the AMA about the emerging landscape of augmented intelligence in health care (PDF).
Stay up to date on AI
Follow the latest news on AI and its applications and effects for health care—delivered to your inbox.
How to evaluate them
How to evaluate them
Using a hypothetical case of a 67-year-old man admitted with a chronic obstructive pulmonary disease exacerbation, the module provides a critical appraisal of a machine learning model that provided an alert that the patient was at risk of clinical deterioration in the next 12 hours.
The critical appraisal process begins with applying a real-world machine learning prognosis study to the case. The module recommends answering the following questions about the study’s participants and predictors/prognostic factors. Each question is supported by a robust explanation.
- What data sources were used?
- Was a defined, representative sample of patients (data) assembled?
- Were the inclusion and exclusion criteria appropriate?
- Were predictors defined and assessed in a similar way for all participants?
- Were predictor assessments made without knowledge of outcome data?
From there, the module advises evaluating outcome criteria, follow-up and subgroups by answering:
- Were outcomes of interest defined in advance? Were there clear criteria to determine if the outcome occurred?
- Were outcomes measured without knowledge of predictor/prognostic factors and clinical characteristics (blindly)?
- Was the time interval between predictor assessment and outcome determination complete?
- Was follow-up sufficiently long for the outcome to occur? Was the follow-up complete?
- Were subgroups with different prognoses identified? If so, was there adjustment for important predictors or prognostic factors?
The module then outlines how to scrutinize the results of the study, as well as how to evaluate the model itself. Periodic knowledge checks and review sections test the user’s vocabulary and their understanding of how concepts are applied.
The CME module “AI for Prognostication and Treatment” is enduring material and designated by the AMA for a maximum of 0.5 AMA PRA Category 1 Credit™.
It is part of the AMA Ed Hub, an online platform with high-quality CME and education that supports the professional development needs of physicians and other health professionals. With topics relevant to you, it also offers an easy, streamlined way to find, take, track and report educational activities.
Learn more about AMA CME accreditation.
