GenAIMeta: Generative AI CDC Metadata Query Application
Description
Metadata plays a crucial role in enhancing public understanding and usage of CDC data. Usable metadata are essential not only for making data easy to find, understand, and use on data.cdc.gov but also for synchronizing with other federal catalogs. Metadata on data.cdc.gov spans 1,056 datasets with ~20 fields each, syncing nightly with federal catalogs. Manual validation, normalization and monitoring of this volume, and the inconsistent quality and completeness of those fields creates bottlenecks for data discovery, governance and downstream analytics. The aim is to leverage EDAV’s Azure OpenAI-powered models to automate metadata validation, standardization and monitoring at scale, replacing error-prone manual checks with real-time, AI-driven oversight.
Detailed example
Phase -1 Implementation ? EDAV’s Azure AI Infrastructure • Ingest and preprocess data from data.cdc.gov • Extract vector embeddings for model training • Build and fine-tune LLM and ML models focused on metadata usage and quality monitoring ? Monitoring Dashboard • Connects directly to Azure AI outputs • Provides real-time data-quality checks and metadata health metrics • Features interactive interfaces for key metrics and insights Phase-2 Implementation ? Domain-Specific Model Training • Built a targeted dataset of real-world questions with paired ideal answers • Fine-tuned and evaluated models against accuracy, relevance, clarity and completeness benchmarks ? Multi-User, Multi-Agent Framework • Deployed specialized agents for distinct roles • Enabled simultaneous support for diverse users including data-quality managers, data scientists, epidemiologists, etc. ensuring scalable, task-focused collaboration
AI / analytics pattern
Generative AI: AI that generates new or synthetic content (e.g., images, videos, audio, text, code).
Automation level / stage
b) Pilot – The use case has been deployed in a limited test or pilot capacity.
Expected benefit
Objective: Phase 1- Automate metadata validation and monitoring on data.cdc.gov using EDAV’s Azure OpenAI API Phase 2 - Make useful AI agents that are user centric and make application of the tool broader with tested agent efficacy Evaluation: Approach: Asked both general and domain-trained models the same set of real-world questions. Benchmarked responses on four metrics: Accuracy: Match to ideal answers Relevance: Alignment with user needs Clarity: Readability and actionability Completeness: Coverage of all aspects of the question Results: Domain-trained models outscored general models on every metric. Trained models delivered more precise, context-aware, and fully-formed answers. General models tended toward vague or overly broad responses. Conclusion: Targeted, domain-specific training significantly boosts an LLM’s ability to meet specialized user requirements. Key Benefits: Actionable insights for better decision-making during time-sensitive scenarios. Optimized resource allocation for improved efficiency. Enhanced trust in decision-making frameworks through consistent performance.
Controls / human review
ATO: Yes; PIA: Not published