HARMONIC is a framework for generating and evaluating synthetic tabular data with large language models (LLMs), with a focus on privacy protection. Instead of relying on continued pre-training of small LLMs, HARMONIC fine-tunes larger LLMs on an instruction dataset built with a k-nearest-neighbors idea, so that the model learns the format of and relationships between rows rather than memorizing the data itself. The evaluation framework adds privacy-risk metrics tailored to LLM-generated data, as well as performance metrics for downstream LLM tasks. Experiments show that HARMONIC matches existing methods in performance while offering better privacy.
Abstract
Data serves as the fundamental foundation for advancing deep learning, particularly tabular data presented in a structured format, which is highly conducive to modeling. However, even in the era of LLM, obtaining tabular data from sensitive domains remains a challenge due to privacy or copyright concerns. Hence, exploring how to effectively use models like LLMs to generate realistic and privacy-preserving synthetic tabular data is emergent. In this paper, we take a step forward to explore LLMs for tabular data synthesis and privacy protection, by introducing a new framework HARMONIC for tabular data generation and evaluation. In our tabular data generation framework, unlike previous small-scale LLM-based methods that rely on continued pre-training, we explore the larger-scale LLMs with fine-tuning to generate tabular data and enhance privacy. Based on idea of the k-nearest neighbors algorithm, an instruction fine-tuning dataset is constructed to inspire LLMs to discover inter-row relationships. Then, with fine-tuning, LLMs are trained to remember the format and connections of the data rather than the data itself, which reduces the risk of privacy leakage. In our evaluation framework, we develop specific privacy risk metrics for LLM synthetic data generation, as well as performance evaluation metrics for downstream LLM tasks. Our experiments find that this tabular data generation framework achieves equivalent performance to existing methods with better privacy, which also demonstrates our evaluation framework for the effectiveness of synthetic data and privacy risks in LLM scenarios.
Table 1. Machine learning efficiency (MLE) and large language model efficiency (LLE)
| Dataset | Metric | Original | OurModel | Smote | TVAE | CTAB | TabDDPM | TABSYN | GReaT | REaLTabFormer |
| GM | MLE | 0.50±0.00 | 0.55±0.03 | 0.64±0.02 | 0.61±0.02 | 0.57±0.02 | 0.64±0.01 | 0.63±0.02 | 0.44±0.03 | 0.65±0.01 |
| LLE | 0.71±0.00 | 0.64±0.03 | 0.67±0.03 | 0.69±0.03 | 0.71±0.02 | 0.67±0.05 | 0.72±0.02 | 0.55±0.11 | 0.69±0.03 | |
| AD | MLE | 0.61±0.00 | 0.67±0.02 | 0.75±0.00 | 0.74±0.00 | 0.73±0.01 | 0.74±0.00 | 0.73±0.01 | 0.73±0.01 | 0.76±0.00 |
| LLE | 0.81±0.00 | 0.80±0.02 | 0.84±0.01 | 0.83±0.01 | 0.83±0.00 | 0.83±0.00 | 0.81±0.02 | 0.82±0.02 | 0.85±0.00 | |
| DI | MLE | 0.56±0.00 | 0.46±0.02 | 0.72±0.03 | 0.71±0.02 | 0.67±0.02 | 0.71±0.02 | 0.68±0.03 | 0.45±0.03 | 0.66±0.03 |
| LLE | 0.70±0.00 | 0.75±0.00 | 0.69±0.04 | 0.72±0.04 | 0.62±0.09 | 0.72±0.03 | 0.77±0.01 | 0.71±0.03 | 0.70±0.04 | |
| BU | MLE | 0.38±0.00 | 0.27±0.03 | 0.25±0.02 | 0.27±0.03 | 0.26±0.01 | 0.27±0.01 | 0.26±0.01 | 0.24±0.03 | 0.26±0.00 |
| LLE | 0.88±0.00 | 0.82±0.03 | 0.85±0.04 | 0.86±0.01 | 0.82±0.02 | 0.85±0.01 | 0.86±0.01 | 0.81±0.03 | 0.70±0.14 |
Table 2. Mean Distance to Closest Record (DCR), New Row Synthesis (NRS) and PPL-diff (PPL)
| Dataset | Metric | OurModel | Smote | TVAE | CTAB | TabDDPM | TABSYN | GReaT | REaLTabFormer |
| GM | NRS | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| DCR | 8.08 | 2.77 | 4.09 | 5.36 | 2.21 | 3.98 | 5.84 | 4.60 | |
| PPL | -0.16 | — | — | — | — | — | -2.14 | -22.04 | |
| AD | NRS | 1.00 | 0.95 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| DCR | 2.47 | 0.16 | 0.49 | 0.82 | 0.50 | 0.86 | 1.51 | 0.57 | |
| PPL | -0.98 | — | — | — | — | — | -0.67 | -163.71 | |
| DI | NRS | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| DCR | 0.44 | 0.28 | 0.33 | 0.72 | 0.21 | 1.37 | 1.36 | 0.36 | |
| PPL | -0.37 | — | — | — | — | — | -0.44 | -42.46 | |
| BU | NRS | 1.00 | 0.93 | 1.00 | 1.00 | 0.99 | 1.00 | 1.00 | 1.00 |
| DCR | 2.52 | 0.15 | 0.66 | 0.70 | 0.18 | 1.38 | 8.30 | 0.38 | |
| PPL | -0.34 | — | — | — | — | — | -2.22 | -41.13 |
The pipeline has four steps: preprocess the data, fine-tune the generator, sample synthetic data, and evaluate it. Scripts live in scripts/; datasets live in Data/.
Template and example:
python scripts/preprocess_data.py [data_name] [seed] [knn_n] [task_type] [des] [re_format] [sample_num]
python scripts/preprocess_data.py german 416 5 "binary classification" "user credit scores" dict 700Template and example:
sh scripts/sft_gen.shTemplate and example:
python scripts/sample.py [data_name] [sample_num] [seed] [temperature] [max_length] [task_type] [device]
python scripts/sample.py german 700 2416 0.7 2048 'binary classification' 'cuda:0'Template and example:
sh scripts/sft_lle.sh
sh scripts/eval-llama2.shIf you find this work useful, please cite:
@misc{wang2024harmonic,
title = {HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection},
author = {Yuxin Wang and Duanyu Feng and Yongfu Dai and Zhengyu Chen and Jimin Huang and Sophia Ananiadou and Qianqian Xie and Hao Wang},
year = {2024},
eprint = {2408.02927},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2408.02927}
}The code in this repository is released under the MIT License. Datasets and models on Hugging Face keep their own licenses, stated on each card.
Built by The Fin AI · Hugging Face · GitHub