Can Synthetic Data Replace Real Data? Evaluating the Efficacy, Limits, and Label Trustworthiness of Generative Data in Supervised Learning
DOI:
https://doi.org/10.53799/sa1d7q88Keywords:
Class imbalance, CTGAN, healthcare machine learning, label corruption, SLTE framework, supervised learning, synthetic data, TSTR protocolAbstract
Synthetic data are increasingly proposed as substitutes for real training records where privacy restricts data sharing. We evaluate this substitution on a medical appointment no-show dataset of 110,526 records with the Train-Synthetic-Test-Real protocol, five classifiers, six synthetic mixing ratios and nine generation runs of CTGAN, TVAE and a Gaussian copula on this and a second dataset (UCI Adult). Full substitution lowered mean AUC-ROC in every run (by 5.72% to 11.46% across classifiers in the primary run), and mixed training beat training on its real part alone in at most six of 40 paired DeLong comparisons per run (Holm correction) while losing in 6 to 30. We propose Synthetic Label Trustworthiness Estimation (SLTE), an empirical label-consistency check that scores each synthetic label against a model trained on real data, relabels low-confidence samples by nearest-neighbour vote and down-weights uncertain ones. SLTE changed 12.88% of the primary synthetic labels, 94.2% of them from minority to majority class, but a label-permutation null reproduced this asymmetry and the same procedure changed 14.20% of held-out real labels, so the changes reflect disagreement with a real-data model rather than generator errors. In the primary run SLTE improved AUC-ROC significantly only at 80% and 100% synthetic data, recovering 57.8% of Gradient Boosting's loss at full substitution, but it lowered recall for most classifiers, and in other runs it lowered AUC-ROC for Decision Tree, Logistic Regression or Naive Bayes. Synthetic data did not replace real data here; SLTE is a quality-control step for when substitution cannot be avoided.
References
[1] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2014.
[2] L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni, “Modeling tabular data using conditional GAN,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2019.
[3] N. Patki, R. Wedge, and K. Veeramachaneni, “The synthetic data vault,” in Proc. IEEE Int. Conf. Data Sci. Adv. Anal. (DSAA), 2016, pp. 399–410. DOI: https://doi.org/10.1109/DSAA.2016.49
[4] A. Kotelnikov, D. Baranchuk, I. Rubachev, and A. Babenko, “TabDDPM: Modelling tabular data with diffusion models,” in Proc. Int. Conf. Mach. Learn. (ICML), 2023.
[5] J. Jordon, J. Yoon, and M. van der Schaar, “PATE-GAN: Generating synthetic data with differential privacy guarantees,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2019.
[6] Z. Zhao, A. Kunar, R. Birke, and L. Y. Chen, “CTAB-GAN: Effective table data synthesizing,” in Proc. Asian Conf. Mach. Learn. (ACML), 2021.
[7] N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari, “Learning with noisy labels,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2013.
[8] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 1944–1952. DOI: https://doi.org/10.1109/CVPR.2017.240
[9] H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, and Y. Liu, “Learning with instance-dependent label noise: A sample sieve approach,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2021.
[10] E. Choi, S. Biswal, B. Malin, J. Duke, W. F. Stewart, and J. Sun, “Generating multi-label discrete patient records using generative adversarial networks,” in Proc. Mach. Learn. Healthcare Conf. (MLHC), 2017.
[11] A. Yale, S. Dash, R. Dutta, I. Guyon, A. Pavao, and K. P. Bennett, “Generation and evaluation of privacy preserving synthetic health data,” Neurocomputing, vol. 416, pp. 244–255, 2020. DOI: https://doi.org/10.1016/j.neucom.2019.12.136
[12] A. Gonzales, G. Guruswamy, and S. R. Smith, “Synthetic data in health care: A narrative review,” PLOS Digit. Health, vol. 2, no. 1, 2023. DOI: https://doi.org/10.1371/journal.pdig.0000082
[13] N. Japkowicz and S. Stephen, “The class imbalance problem: A systematic study,” Intell. Data Anal., vol. 6, no. 5, pp. 429–449, 2002. DOI: https://doi.org/10.3233/IDA-2002-6504
[14] M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, “Do we need hundreds of classifiers to solve real world classification problems?” J. Mach. Learn. Res., vol. 15, no. 1, pp. 3133–3181, 2014.
[15] R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,” Inf. Fusion, vol. 81, pp. 84–90, 2022. DOI: https://doi.org/10.1016/j.inffus.2021.11.011
[16] L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why tree-based models still outperform deep learning on tabular data,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2022. DOI: https://doi.org/10.52202/068431-0037
[17] J. Aquino, “Medical appointment no show,” Kaggle, 2016. [Online]. Available: https://www.kaggle.com/datasets/joniarroba/noshowappointments
Downloads
Published
Data Availability Statement
The dataset analysed in this study is publicly available. The Medical Appointment No-Show dataset can be accessed at https://www.kaggle.com/datasets/joniarroba/noshowappointments. All preprocessing steps, CTGAN hyperparameters, and classifier configurations required to reproduce the results are fully specified in Section III of the manuscript. No new data were generated beyond the synthetic records described, which can be regenerated using the stated parameters and random seed.
Issue
Section
License
Copyright (c) 2026 AIUB Journal of Science and Engineering (AJSE)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
AJSE contents are under the terms of the Creative Commons Attribution License. This permits anyone to copy, distribute, transmit and adapt the work non-commercially provided the original work and source is appropriately cited.
