Use of Large Language Models to improve classification of identified text snippets from the CLEVER natural language processing pipeline
Description
Use of text matching approaches like CLEVER to identify clinical information of interest in free-text clinical notes is an efficient approach but can be over-inclusive (e.g. including terms when used with an alternative meaning, for example, a search for “arms” may pick up mentions of forearms and firearms). A variety of approaches can be used to further filter identified text snippets to those more likely to be of interest for a given use case. Here we use large language model queries to classify text snippets as being relevant to a concept of interest.
Detailed example
This AI use case adds additional LLM-derived classification labels to text snippets from free-text clinical notes identified by the CLEVER NLP pipeline. These labels indicate the LLMs classification of the text snippet as being related to a concept of interest. For example, in our first deployed use of this method, an LLM model was used to improve classification of text snippets identified as including the term “xylazine” in the note text. The LLM classified identified text snippets into one of three labels: (1) "Other" (OTH) for snippets without evidence of suspected xylazine exposure, (2) "Suspected-Positive" (SUS-P) for snippets with positively asserted evidence of suspected exposure, and (3) "Suspected-Negative" (SUS-N) for snippets where the suspected exposure was negated. The OTH category included, for example, the many cases in which providers educated patients about the presence of xylazine in the illicit drug supply. The LLM-derived classification labels were used to narrow down text snippets for display as mentions of “possible xylaxine exposure” on the STORM decision support display (to help clinicians find information of relevance to opioid risk management across VA medical records) and in maps of possible xylazine exposures for strategic planning. In both cases, only those text snippets labeled as “SUS-P” were included on the displays.
AI / analytics pattern
Natural Language Processing: AI that processes, interprets, and shares information in human language.
Automation level / stage
c) Deployed – The use case is being actively authorized or utilized to support the functions or mission of an agency.
Expected benefit
Improved classification of text snippets from clinical notes will increase clinician and management efficiency in use of decision support to summarize and review information extracted from free-text clinical notes. For example, improved filtering of text snippets for clinical risk factors of interest will reduce clinician time in obtaining clinically needed information about risk factors of interest by filtering out text snippets that are unlikely to be meaningful to their immediate clinical decision. In short, the clinician will have fewer candidate text snippets to read through to get the information they need. Likewise, summary views of identified possible concept mentions will be more accurate, reducing noise in views to support strategic decisions.
Controls / human review
ATO: Not reported; PIA: Not published