Synthetic healthcare data can appear realistic across a whole population while failing to reproduce important patient subgroups. A recent analysis in BMC Medical Informatics and Decision Making assessed quality measures for synthetic data using lung cancer registry data from the Netherlands. Synthetic data can support research when access to patient-level information is limited by privacy requirements or data scarcity, but its value depends on whether relevant patterns are preserved. The findings show that strong overall fidelity and utility scores can hide subgroup differences, false correlations and changes in predictive performance, especially in small patient groups.
Why Aggregate Quality Scores Can Mislead
Quality assessment of synthetic healthcare data needs to consider both realism and privacy. Realism measures test whether synthetic records preserve important statistical patterns and whether models trained on one version of the data perform well on another. Privacy measures assess risks such as identifying whether a person was in the original dataset or inferring sensitive information. These aims can conflict because stronger privacy protection may require added noise or changes to the data structure, which can reduce analytical usefulness.
No single metric can capture every requirement. Synthetic data may preserve broad distributions while losing more complex relationships, rare patterns or clinically meaningful correlations. Generative models often reproduce dominant patterns and average trends better than unusual or less common cases. This can be important when small subgroups contain clinically relevant characteristics, such as atypical treatment responses or demographic patterns. Different metrics can also produce different conclusions because they measure separate aspects of data quality.
The lung cancer evaluation therefore compared full-cohort and subgroup performance rather than relying only on global scores. It used three synthetic data generators and assessed both fidelity and cross-classification utility. Models were trained and tested across real and synthetic datasets to examine whether predictive signals were preserved. Survival prediction was evaluated with logistic regression, a non-linear support vector machine and a gradient boosting classifier.
Subgroup Testing Reveals Performance Gaps
The main case included 50,000 patients with non-small cell lung cancer registered in the Netherlands Cancer Registry between 2014 and 2019. A clinically defined subgroup included 992 patients with stage IV disease who were ineligible for surgery or radiotherapy and received systemic chemotherapy or immunotherapy. The same subgroup criteria were applied to real and synthetic data. The survival models used the same 22 selected predictors in both the full cohort and the subgroup.
Across the full cohort, the synthetic datasets achieved high fidelity scores. Predictive performance was also relatively similar between models trained on real and synthetic data in several comparisons. However, visual assessment found an abnormal relationship between Eastern Cooperative Oncology Group performance status and survival in the synthetic data. This showed that strong overall scores did not preserve every clinically relevant association. The synthetic datasets also showed less variability than the real data in some model results.
The differences became clearer within the stage IV subgroup. With PrivBayes, logistic regression accuracy fell from 69% when trained and tested on real data to 52% when trained on real data and tested on synthetic data. Other configurations produced unusually high synthetic-only performance, including 88% with the non-linear support vector machine. The gradient boosting classifier showed a similar pattern. Comparable subgroup problems were seen with the other generators.
Subgroup-Aware Evaluation Is Needed
Feature importance also changed between real and synthetic subgroup data. In the full cohort, four of the five most important survival predictors were shared between real and PrivBayes-generated data. In the stage IV subgroup, however, the synthetic data gave greater importance to different variables, including an ECOG category linked to the abnormal survival pattern identified in the qualitative assessment. Similar inconsistencies appeared with the other generation methods.
Evaluation should therefore include clinically meaningful subgroups instead of treating overall fidelity as sufficient evidence of quality. Synthetic generation can reduce variability and smooth complex relationships, producing datasets that look convincing at population level while losing distinctions needed for specific analyses. Clinical decisions or actions should not be based only on synthetic data, and findings should later be checked against real data. Subgroup checks can be carried out before synthetic data are released for wider use.
Synthetic data can still be useful for tasks where very high realism or privacy requirements are not essential. These include prototype development, feature engineering, data augmentation, clinical trial site feasibility work, harmonisation and mapping to common data models and education. However, assessment should match the intended use, and relevant patient groups should be tested before synthetic data are used for research, reporting or regulatory purposes. The evaluation also had limitations, including a restricted set of utility measures and a limited number of case studies and subgroup definitions.
Synthetic data can help address limits on healthcare data access and privacy, but overall quality scores may give incomplete reassurance. In the lung cancer evaluation, strong full-cohort fidelity existed alongside subgroup changes in predictive performance, feature importance and clinical relationships. The findings support assessment that combines fidelity and utility measures with subgroup analysis and qualitative clinical checks. They also support generation methods that account directly for rare or minority groups. Until these approaches improve, synthetic data require cautious use when analyses depend on detailed subgroup representation or complex clinical relationships.
Source: BMC Medical Informatics & Decision Making
Image capture:
Internal link: https://healthmanagement.org/c/it/News/synthetic-data-supports-cancer-survival-model-transfer
References:
Chatzichristos C, Katsimpokis D, Lai C et al. (2026) Quality assessment of synthetic data in healthcare: a critical appraisal with a focus on subgroup bias. BMC Med Inform Decis Mak. https://doi.org/10.1186/s12911-026-03719-3
