Open-source topic identification for the Public Engagement with Science Initiative data analysis.
Description
OIA team proposes using unsupervised Natural Language Processing (NLP) to identify topical themes within responses submitted to the NSF public engagement questionnaire. The AI system will include multiple open-source techniques used together to achieve this goal. Details of the system are as follows: To ensure high-quality topics, the system will first programmatically remove responses that are blank or non-informative. The National Center for Health Statistics (NCHS) Semi-Automated Non-Response Detection for Surveys (SANDS) model will be used to filter out non-response responses (e.g., ""sdfsdfa,"" "" idkkkkk""). To reduce duplicate responses potentially caused by bots or submitted to boost relative importance, responses with the exact duplicate question wording will be removed over a threshold of three responses per IP address. If an individual has several different questions they would like to submit, all of their questions will be included. The system will use an open-source profanity filter such as Python’'s profanity-filter to identify toxic language. Any responses that are primarily toxic language will be dropped from analysis. To group questions into themes, we will use a word-embedding model to convert text responses into multidimensional space for topic modeling. Because we anticipate responses in multiple languages, the system will use a multilingual will use a multilingual word-embedding model, such as "paraphrase-multilingual-MiniLM-L12-v2" or the most up-to-date version of this model at the time of analysis. Embeddings will be reduced in dimensions using a dimension reduction algorithm such as t-SNE (t-distributed Stochastic Neighbor Embedding) to prepare for clustering. Reduced response embeddings will be clustered into groups using the ClassTfidfTransformer clustering algorithm from the Python package BertTopic. The algorithm will be seeded with words corresponding to a list of NSF programs corresponding to different research areas to map the clusters to these established categorizations. To aid in the review of topics generated by the clustering algorithm, an open-source large language model (LLM), such as Meta Llama, will be used to summarize the responses grouped within a cluster. Using a prompt like the example below, this strategy will produce a plain language summary and title of each of the clusters of responses. The clusters produced will be reviewed by Research Triangle Institute (RTI) staff for accuracy and appropriateness of naming. Staff will make edits to the LLM-generated summaries as necessary and note any clusters that are candidates for merging with other clusters into a single topic. The topics will then be validated by NSF experts and the advisory committee. Once the existing cluster model has been finalized, the cluster model will be applied to all responses. Where feasible, the RTI Science, Technology, Engineering, and Mathematics (STEM) education experts/analysts will manually map the research topics to NSF research programs.
Automation level / stage
a) Pre-deployment – The use case is in a development or acquisition status.
Controls / human review
ATO: Not reported; PIA: Not published