The Most Conservative Test We Could Run
When we set out to validate Simsurveys against the Kaiser Family Foundation’s July 2023 Health Tracking Poll on GLP-1 weight loss drugs, we deliberately chose the hardest possible conditions. We used our untrained consumer model — no disease-specific training data, no custom quotas, no targeting parameters. We matched the KFF’s sample size of 1,327 U.S. adults drawn from a general population frame. If the model could perform well under these constraints, it would provide strong evidence that synthetic survey data can hold up even without domain-specific fine-tuning.
The KFF Health Tracking Poll, fielded July 11–19, 2023, used a probability-based panel to survey American adults on their awareness of GLP-1 drugs like Ozempic, Wegovy, and Mounjaro; their willingness to try these medications; their trust in pharmaceutical companies; and their views on drug pricing and insurance coverage. The poll captured a moment of extraordinary public attention to GLP-1 drugs, making it a high-stakes benchmark for any model claiming to replicate consumer health attitudes.
Overall Performance
Across 20 questions organized into five thematic clusters, the synthetic model achieved an average KL divergence of 0.039 and a median of 0.033. Of the 20 questions, 12 scored below 0.05 (“Strong” alignment), 8 scored between 0.05 and 0.15 (“Good” alignment), and zero required review. This is one of the strongest overall validation results in our portfolio, and it was achieved under the most conservative testing conditions — no model customization whatsoever.
The five question clusters covered Drug Awareness (2 questions), Willingness to Try (5 questions), Trust in Pharma (5 questions), Affordability & Adherence (5 questions), and Insurance & Pricing (3 questions). Performance was consistent across clusters, with no single domain showing systematic weakness.
Summary: Average KL divergence 0.039, median 0.033. 12 of 20 questions “Strong” (<0.05), 8 of 20 “Good” (0.05–0.15), 0 needing review. Achieved with an untrained general-population model — no disease-specific training, no custom quotas, no targeting.
Near-Perfect Alignment on Consensus Items
Some of the most striking results came on questions where public opinion is strongly directional. On the question of whether GLP-1 drug costs are unreasonable, 82% of KFF respondents agreed — and 82% of synthetic respondents agreed, producing a KL divergence of 0.000. On the question of whether pharmaceutical companies prioritize profit over patients, 83% of real respondents agreed versus 78% of synthetic respondents, for a KL divergence of just 0.011.
These consensus items demonstrate that the model reliably captures settled public sentiment. When Americans broadly agree on something — that drug prices are too high, that pharma companies are profit-driven — the synthetic model reflects that consensus with near-zero error. This is not trivial; it means the model has internalized the dominant narratives in American health policy discourse.
Four Systematic Patterns
While the overall results were strong, a careful question-by-question analysis revealed four systematic patterns in how the synthetic model deviated from the KFF benchmark.
GLP-1 awareness inflation. The synthetic model slightly overestimated public awareness of GLP-1 drugs compared to the July 2023 KFF data. This is almost certainly a temporal effect: the model’s training data reflects 2025 media coverage and public discourse, during which GLP-1 awareness increased dramatically. The KFF poll captured a snapshot from mid-2023, when drugs like Ozempic were well-known but Mounjaro and Zepbound had not yet achieved the same cultural penetration. For researchers using the model today, the awareness levels may actually be more current than the 2023 benchmark.
Cost and adherence pessimism. On questions about medication affordability and likelihood of continued adherence, the synthetic model showed slightly more pessimistic views than the KFF sample. Synthetic respondents were marginally more likely to say they could not afford GLP-1 drugs and marginally less confident about long-term adherence. This may reflect the intensification of public debate about drug pricing in the intervening period.
Moderate pharma trust skepticism. The model showed slightly elevated skepticism toward pharmaceutical companies on trust questions, consistent with the general pattern of the model reflecting somewhat more pronounced or crystallized versions of prevailing public sentiment.
Near-perfect on consensus items. As noted above, questions where public opinion is strongly directional produced the tightest alignment, often with KL divergences below 0.01. The model excels at reproducing settled attitudes and struggles most with questions where opinion is genuinely divided or rapidly evolving.
What This Means for Pharma Research
The KFF validation carries particular weight because of the testing conditions. An untrained, general-population consumer model — with no GLP-1-specific training data, no patient targeting, and no custom demographic quotas — achieved an average KL divergence of 0.039 across 20 questions on one of the most actively discussed pharmaceutical topics in America. For pharma companies, payers, and health policy researchers, this suggests that synthetic survey data can reliably capture public attitudes toward drugs, pricing, and healthcare access even without disease-specific model customization.
For studies requiring even higher precision, the Simsurveys Patient model — which is trained on federal health data and supports condition-based targeting — would be expected to perform even better on questions about specific therapeutic areas, treatment experiences, and patient-reported outcomes.
The full validation report, including question-level distribution tables and cluster-by-cluster analysis, is available for download on our validation studies page. To get started with your own pharma or patient research study, create a free account.