AI Data Generator

A powerful, flexible AI-driven data generation tool for Databricks that uses foundational models to create realistic sample data for any industry and domain.

📁 Folder Structure

dbx_ai_synth_data/
├── ai_data_generator.py          # Main notebook/script
│
├── bundle/                        # Databricks Asset Bundle
│   ├── databricks.yml            # Main bundle configuration (with sync)
│   ├── job_parameters.conf       # Parameter templates
│   ├── README.md                 # Bundle-specific documentation
│   └── resources/
│       └── jobs.yml              # Job definitions (4 pre-configured jobs)
│
└── docs/                         # Documentation
    ├── QUICKSTART.md             # 3-minute getting started
    ├── EXAMPLES.md               # 19+ usage examples
    ├── JOB_SUMMARY.md            # Job deployment overview
    ├── DEPLOYMENT.md             # Complete deployment guide
    ├── QUICK_REFERENCE.md        # Command reference card
    └── ARCHITECTURE.md           # Technical architecture

Key Features:

• ✅ Automatic File Sync: ai_data_generator.py automatically syncs to workspace during deployment
• ✅ Relative Paths: Jobs use ${workspace.file_path} for environment-agnostic deployment
• ✅ Multi-Environment: Dev, staging, and prod configurations included

🚀 Quick Start

Option 1: Run as Notebook (Simple)

1. Upload ai_data_generator.py to Databricks workspace
2. Open in notebook and configure parameters
3. Run all cells

Option 2: Deploy as Job (Recommended)

Before deploying, you MUST update the workspace URL in bundle/databricks.yml:

# 1. Navigate to bundle directory
cd bundle

# 2. IMPORTANT: Update workspace URL in databricks.yml (line 42)
# Edit: host: "https://your-actual-workspace.cloud.databricks.net/"
# Replace with your actual Databricks workspace URL

# 3. Configure authentication
databricks configure --token

# 4. Validate bundle
databricks bundle validate -t dev

# 5. Deploy (automatically syncs ai_data_generator.py to workspace)
databricks bundle deploy -t dev

# 6. Run
databricks bundle run generate_patients_job -t dev

📖 Documentation Guide

Document	Purpose	Start Here If...
docs/QUICKSTART.md	3-minute start	You want to run it quickly
docs/JOB_SUMMARY.md	Job deployment overview	You want to deploy as a job
docs/DEPLOYMENT.md	Full deployment guide	You need detailed deployment steps
docs/QUICK_REFERENCE.md	Command reference	You need quick command lookup
docs/EXAMPLES.md	19+ templates	You need example configurations
docs/ARCHITECTURE.md	Technical details	You want deep technical knowledge
bundle/README.md	Bundle configuration	You want bundle deployment details

✨ Key Features

• AI-Generated Schemas: Let AI create appropriate schemas for your industry
• Custom Schemas: Or provide your own schema for full control
• Column Constraints: Fine-tune data generation per column (e.g., "price between 10-500")
• Industry-Aware: Data is contextually realistic for your specific industry
• Job Deployment: Deploy as Databricks jobs with full automation
• Multi-Environment: Dev, staging, production support
• Multi-Table: Generate related tables with dependencies

🎯 Use Cases

• Testing: Generate test data for development pipelines
• Demos: Create realistic demo data for presentations
• POCs: Quick data generation for proof-of-concepts
• Training: Generate sample datasets for training
• Development: Match production schemas in dev environments

📋 Configuration Parameters

Parameter	Description	Example
industry	Industry domain	"healthcare", "retail", "finance"
domain	Data domain	"patient records", "transactions"
table_name	Output table name	"patients", "products"
num_rows	Number of rows	"100", "1000", "10000"
ai_model_endpoint	AI model to use	"databricks-meta-llama-3-3-70b-instruct"
custom_schema_json	Optional JSON schema	'{"col": "TYPE"}'
column_constraints_json	Optional constraints	'{"col": "rule"}'

🎓 Example Usage

Healthcare Patient Records

databricks bundle run ai_data_generator_job -t dev \
  --param industry="healthcare" \
  --param domain="patient records" \
  --param table_name="patients" \
  --param num_rows="500"

Retail Product Inventory

databricks bundle run ai_data_generator_job -t dev \
  --param industry="retail" \
  --param domain="product inventory" \
  --param table_name="products" \
  --param num_rows="200"

Finance Transactions

databricks bundle run ai_data_generator_job -t dev \
  --param industry="finance" \
  --param domain="transactions" \
  --param table_name="transactions" \
  --param num_rows="1000"

🔧 Pre-Configured Jobs

The bundle includes 4 ready-to-use jobs:

1. ai_data_generator_job - Generic template (fully customizable)
2. generate_patients_job - Healthcare patients (50,000 rows)
3. generate_products_job - Retail products (20,000 rows)
4. generate_transactions_job - Finance transactions (10,000 rows)

📊 Performance

• ~2-3 seconds per row for AI generation
• Recommended: 50-200 rows for testing
• Production: Scale with cluster size and batch processing

Rows	Cluster Workers	Time Estimate
10-50	2	1-3 minutes
100-500	2	3-10 minutes
1,000	2-4	10-30 minutes
10,000+	8+	1+ hours

🌍 Multi-Environment Support

Deploy to different environments with environment-specific configurations:

# Development
databricks bundle deploy -t dev

# Staging
databricks bundle deploy -t staging

# Production
databricks bundle deploy -t prod

🐛 Troubleshooting

Having issues? Check:

1. bundle/README.md - Bundle-specific deployment guide
2. docs/DEPLOYMENT.md - Detailed deployment guide
3. docs/JOB_SUMMARY.md - Job configuration overview

Common fixes:

• Update workspace URL in bundle/databricks.yml (line 42) - THIS IS REQUIRED!
• Configure authentication: databricks configure --token
• Validate bundle: databricks bundle validate -t dev
• Verify cluster variables are defined (or use serverless compute)

🔗 Links

• Documentation: See docs/ folder
• Bundle Configuration: See bundle/ folder (⚠️ Update workspace URL before deploying!)
• Parameter Templates: See bundle/job_parameters.conf
• Databricks Bundles: Official Documentation

Important: Workspace Configuration

Before deploying, you MUST update the workspace host in bundle/databricks.yml:

workspace:
  host: "https://your-actual-workspace.cloud.databricks.net/"

The bundle automatically syncs ai_data_generator.py from the root directory to your workspace during deployment.

📞 Getting Help

1. Quick Start: Read docs/QUICKSTART.md
2. Bundle Setup: Check bundle/README.md for workspace configuration
3. Examples: Check docs/EXAMPLES.md for 19+ templates
4. Full Guide: Read docs/DEPLOYMENT.md

Before You Start

⚠️ IMPORTANT: Update bundle/databricks.yml with your workspace URL before deploying!

✅ Prerequisites

• Databricks workspace with Unity Catalog
• Access to AI foundational models
• Databricks CLI (for job deployment)
• Appropriate permissions (create jobs, clusters, tables)

🎉 Quick Commands

# Validate configuration
cd bundle && databricks bundle validate -t dev

# Deploy to dev
databricks bundle deploy -t dev

# Run a job
databricks bundle run generate_patients_job -t dev

# List deployed jobs
databricks bundle jobs list -t dev

# Destroy deployment
databricks bundle destroy -t dev

📝 License

This tool is provided as-is for use with Databricks workspaces that have access to AI foundational models.

---

Version: 1.0
Last Updated: February 2026
Compatible with: Databricks Runtime 13.0+

Ready to generate data? Start with docs/QUICKSTART.md! 🚀