chore: improve engine specific guidance

docs/engines/README.md

@@ -0,0 +1,41 @@
+# Engine documentation
+
+LakeBench engines provide a consistent benchmark interface while preserving each runtime's native storage, authentication, telemetry, and pricing behavior.
+
+| Engine | Install | Primary runtime | Documentation | Coverage |
+|---|---|---|---|---|
+| Generic Spark | `lakebench[spark]` | Local or Spark runtime | [Spark](spark/README.md) | [Report](../../reports/coverage/spark.md) |
+| Fabric Spark | Runtime-provided Spark | Microsoft Fabric | [Fabric Spark](fabric-spark/README.md) | — |
+| Synapse Spark | Runtime-provided Spark | Azure Synapse | [Synapse Spark](synapse-spark/README.md) | — |
+| HDInsight Spark | Runtime-provided Spark | Azure HDInsight | [HDInsight Spark](hdi-spark/README.md) | — |
+| Databricks Spark | `lakebench[databricks_spark]` | Databricks on AWS, Azure, or GCP | [Databricks Spark](databricks-spark/README.md) | — |
+| Databricks SQL Warehouse | `lakebench[databricks_sql_warehouse]` | Databricks SQL | [Databricks SQL Warehouse](databricks-sql-warehouse/README.md) | — |
+| Fabric Data Warehouse | `lakebench[fabric_data_warehouse]` | Microsoft Fabric | [Fabric Data Warehouse](fabric-data-warehouse/README.md) | — |
+| DuckDB | `lakebench[duckdb]` | Local or notebook runtime | [DuckDB](duckdb/README.md) | [Report](../../reports/coverage/duckdb.md) |
+| Polars | `lakebench[polars]` | Local or notebook runtime | [Polars](polars/README.md) | [Report](../../reports/coverage/polars.md) |
+| Daft | `lakebench[daft]` | Local or notebook runtime | [Daft](daft/README.md) | [Report](../../reports/coverage/daft.md) |
+| Sail | `lakebench[sail]` | Local or notebook runtime | [Sail](sail/README.md) | [Report](../../reports/coverage/sail.md) |
+
+All engines write benchmark results through the same shared result contract. A benchmark accepts the engine instance, so the benchmark workflow remains the same after engine construction:
+
+```python
+from lakebench.benchmarks import TPCH
+
+benchmark = TPCH(
+    engine=engine,
+    scenario_name="sf10",
+    scale_factor=10,
+    input_folder_uri="path-or-cloud-uri/tpch/sf10",
+)
+benchmark.run()
+```
+
+## Shared pricing contract
+
+Engines backed by vCore compute can accept either `cost_per_vcore_hour` or `cost_per_hour`; the inputs are mutually exclusive. A total hourly rate should include every cost component the user wants represented in `estimated_retail_job_cost`.
+
+Automatic estimates use public list prices and are not invoice calculations. They do not account for negotiated discounts, commitments, credits, taxes, or other private billing adjustments.
+
+## Adding an engine
+
+Contributor guidance for custom engines remains in the root [README](../../README.md). Internal helpers such as `DeltaRs` do not have standalone engine pages.

docs/engines/daft/README.md

@@ -0,0 +1,36 @@
+# Daft
+
+The `Daft` engine integrates Daft DataFrames with LakeBench and Delta Lake.
+
+## Current support status
+
+Daft is unsupported for ELTBench, TPC-H, and TPC-DS as of LakeBench 2.0.0. The bundled TPC generators emit Parquet containing Arrow `Utf8View`, which the pinned Daft reader cannot load. ClickBench remains partially usable because its sample does not use that generated schema.
+
+See the [Daft coverage report](../../../reports/coverage/daft.md).
+
+## Installation
+
+```bash
+pip install "lakebench[daft]"
+```
+
+## Example
+
+```python
+from lakebench.engines import Daft
+
+engine = Daft(
+    schema_or_working_directory_uri="file:///tmp/lakebench",
+)
+```
+
+## Configuration
+
+- `schema_or_working_directory_uri`: Delta table root.
+- `cost_per_vcore_hour` or `cost_per_hour`: mutually exclusive manual pricing inputs.
+
+For ADLS Gen2, set `AZURE_STORAGE_TOKEN`. OneLake paths are rejected; use an ADLS Gen2 path instead.
+
+## Engine behavior
+
+Daft reads and writes Delta tables through its native Delta integration and the shared DeltaRs helper where appropriate. Native TPC `.tbl` and `.dat` input is implemented, but the generated Parquet limitation currently prevents supported end-to-end TPC/ELT execution.

docs/engines/databricks-spark/README.md

@@ -0,0 +1,68 @@
+# Databricks Spark
+
+`DatabricksSpark` runs LakeBench on Databricks clusters across AWS, Azure, and Google Cloud.
+
+## Support
+
+- Benchmarks: ELTBench, TPC-DS, TPC-H, and ClickBench.
+- Compute: Jobs Compute and All-Purpose Compute.
+- Runtime: Databricks Runtime with PySpark in **Dedicated** access mode.
+- Storage: S3 on AWS, ADLS Gen2 on Azure, and Google Cloud Storage on GCP.
+- Catalogs: Hive metastore or Unity Catalog.
+
+## Installation
+
+```python
+%pip install lakebench
+```
+
+## Example
+
+```python
+from lakebench.engines import DatabricksSpark
+
+engine = DatabricksSpark(
+    catalog_name="main",
+    schema_name="lakebench",
+    schema_uri="abfss://container@account.dfs.core.windows.net/lakebench" #optional
+)
+```
+
+Use `s3://...` on AWS or `gs://...` on GCP.
+
+## Compute and access mode
+
+LakeBench supports both Databricks Jobs Compute and All-Purpose Compute. The compute type is automatically detected and logged in the results.
+
+The compute must use **Dedicated** access mode. LakeBench uses the driver `SparkContext` to set a job description for every timed phase and test item. Those descriptions include labels such as benchmark phase and query ID, making individual operations easy to identify in the Spark UI. Databricks compute modes that do not expose `SparkContext` cannot provide this labeling or the other Spark runtime metadata LakeBench requires.
+
+## Configuration
+
+- `catalog_name`: defaults to `hive_metastore`; use a Unity Catalog name when applicable.
+- `schema_name` and `schema_uri`: benchmark table namespace and Delta root.
+- `cost_per_hour` or `cost_per_vcore_hour`: mutually exclusive manual rates.
+- `spark_measure_telemetry` and `tblproperties`: shared Spark options.
+
+## Pricing
+
+### Azure
+
+For fixed-size clusters, LakeBench combines:
+
+1. Driver and worker DBUs/hour from the published workload-, Photon-, and VM-specific mapping.
+2. The regional DBU price from Azure Retail Prices.
+3. Regional driver and worker VM infrastructure prices from Azure Retail Prices.
+
+Autoscaling or missing cluster metadata causes a warning and leaves cost unset unless a manual rate is supplied. Public prices do not include negotiated discounts or commitments.
+
+Refresh the generated mapping with:
+
+```bash
+uv run python scripts/refresh_databricks_azure_dbu_map.py
+```
+
+### AWS and GCP
+
+Supply an all-inclusive `cost_per_hour` or `cost_per_vcore_hour`. The rate should include both Databricks DBUs and cloud infrastructure. Without a manual rate, the benchmark continues and estimated cost remains unset.
+
+Native TPC `.tbl` and `.dat` input is supported through the shared Spark loader.

docs/engines/databricks-sql-warehouse/README.md

@@ -0,0 +1,60 @@
+# Databricks SQL Warehouse
+
+`DatabricksSQLWarehouse` runs LakeBench through the Databricks SQL connector and Warehouses REST API on AWS, Azure, or GCP.
+
+## Support
+
+- Benchmarks: ELTBench, TPC-DS, TPC-H, and ClickBench.
+- Warehouse types: serverless, classic, and pro.
+- Storage: cloud object storage accessible to the warehouse.
+- Result persistence: SQL executed by the remote warehouse, including result-table appends.
+
+## Installation
+
+```bash
+pip install "lakebench[databricks_sql_warehouse]"
+```
+
+## Example
+
+```python
+import os
+
+from lakebench.engines import DatabricksSQLWarehouse
+
+engine = DatabricksSQLWarehouse(
+    server_hostname="dbc-....cloud.databricks.com",
+    warehouse_name="Serverless Starter Warehouse",
+    catalog_name="main",
+    schema_name="lakebench",
+    access_token=os.environ["DATABRICKS_TOKEN"],
+)
+```
+
+## Configuration
+
+- `server_hostname`: workspace hostname without `https://`.
+- `warehouse_name`: Warehouses API display name.
+- `catalog_name` and `schema_name`: target namespace.
+- `access_token`: used by the REST API and SQL connector; never written to result metadata.
+- `schema_uri`: optional managed/external schema location.
+- `enable_result_caching`: defaults to `False` for reproducible benchmark execution.
+- `cost_per_hour`: all-inclusive manual hourly rate.
+- `dbu_rate`: optional serverless DBU-rate override.
+
+SQL Warehouse intentionally does not accept `cost_per_vcore_hour`.
+
+## Serverless pricing
+
+Warehouse-size DBUs/hour are multiplied by a DBU list rate. Resolution order is:
+
+1. Explicit `dbu_rate`.
+2. Exact recent warehouse SKU from `system.billing.usage`, then its active USD DBU rate from `system.billing.list_prices`.
+3. When no recent usage exists, AWS regional SKU matching if all matching pricing tiers have one rate.
+4. On Azure, the regional `Premium Serverless SQL DBU` meter from the public Azure Retail Prices API.
+
+Insufficient billing-table permissions are logged at warning level. Azure can still use its public catalog; AWS and GCP continue without an estimate unless a manual rate is supplied. Classic and pro warehouses require `cost_per_hour` for estimated cost.
+
+List prices are estimates and do not include negotiated discounts, commitments, or credits.
+
+Native TPC `.tbl` and `.dat` input is loaded with `read_files` and the benchmark DDL.

docs/engines/duckdb/README.md

@@ -0,0 +1,37 @@
+# DuckDB
+
+The `DuckDB` engine provides an embedded local analytical engine with Delta Lake persistence.
+
+## Support
+
+- Benchmarks: ELTBench, TPC-DS, TPC-H, and ClickBench.
+- Storage: local filesystem, mount-style paths, ADLS Gen2, and OneLake with appropriate credentials.
+- Coverage: [DuckDB coverage report](../../../reports/coverage/duckdb.md).
+
+## Installation
+
+```bash
+pip install "lakebench[duckdb]"
+```
+
+## Example
+
+```python
+from lakebench.engines import DuckDB
+
+engine = DuckDB(
+    schema_or_working_directory_uri="file:///tmp/lakebench",
+)
+```
+
+## Configuration
+
+- `schema_or_working_directory_uri`: Delta table root.
+- `storage_options`: options forwarded to DeltaRs and filesystem clients.
+- `cost_per_vcore_hour` or `cost_per_hour`: mutually exclusive manual pricing inputs.
+
+For OneLake `abfss://` paths, set `AZURE_STORAGE_TOKEN`; LakeBench creates a DuckDB Azure secret using that bearer token.
+
+## Engine behavior
+
+DuckDB creates typed empty tables in memory from benchmark DDL and persists them as Delta tables. Delta writes use the shared DeltaRs helper. Native TPC delimited input is supported.

docs/engines/fabric-data-warehouse/README.md

@@ -0,0 +1,52 @@
+# Fabric Data Warehouse
+
+`FabricDataWarehouse` runs LakeBench through the Microsoft Fabric Warehouse SQL endpoint.
+
+## Support
+
+- Benchmarks: ELTBench, TPC-DS, TPC-H, and ClickBench.
+- Runtime: Microsoft Fabric notebook.
+- Storage: OneLake, ADLS Gen2 sources supported by Warehouse SQL, Parquet, and native TPC delimited files.
+- SQL dialect: Fabric Warehouse T-SQL.
+
+## Installation
+
+Use Python 3.10 or later and install:
+
+```python
+%pip install "lakebench[fabric_data_warehouse]"
+```
+
+The host must also have Microsoft ODBC Driver 18 for SQL Server. Pip cannot install this system driver.
+
+## Example
+
+```python
+from lakebench.engines import FabricDataWarehouse
+
+engine = FabricDataWarehouse(
+    warehouse_name="warehouse",
+    warehouse_server="xxxxx.datawarehouse.fabric.microsoft.com",
+    schema_name="dbo",
+)
+```
+
+Warehouse and schema names cannot contain spaces.
+
+## Authentication
+
+The engine uses the current Fabric notebook identity to obtain a Fabric access token. The identity must be able to connect to the Warehouse, create and modify benchmark tables, and read source storage.
+
+## Engine behavior
+
+- ODBC connections use encryption and token authentication.
+- Parquet loads use Warehouse-native SQL paths.
+- TPC native delimited files are loaded with the generator's trailing delimiter handled explicitly.
+- Result-log appends use DeltaRs when required by the result destination.
+- Capacity SKU, Warehouse version, server, Warehouse name, and V-Order state are recorded when visible.
+
+## Pricing
+
+`estimated_retail_job_cost` remains unset. Fabric capacity cost cannot be reliably attributed to an individual Warehouse query or load, so LakeBench does not report a misleading per-job estimate.
+
+`FabricWarehouse` is retained as a compatibility alias for `FabricDataWarehouse`.

docs/engines/fabric-spark/README.md

@@ -0,0 +1,54 @@
+# Fabric Spark
+
+`FabricSpark` specializes the generic Spark engine for Microsoft Fabric notebooks and Lakehouses.
+
+## Support
+
+- Benchmarks: ELTBench, TPC-DS, TPC-H, and ClickBench.
+- Runtime: Microsoft Fabric Spark.
+- Storage: OneLake and Lakehouse schemas.
+- Authentication: the Fabric notebook identity and runtime-provided `notebookutils`.
+
+## Installation
+
+Fabric supplies PySpark. Install LakeBench in the notebook:
+
+```python
+%pip install lakebench
+```
+
+For stage telemetry:
+
+```python
+%pip install "lakebench[sparkmeasure]"
+```
+
+## Example
+
+```python
+from lakebench.engines import FabricSpark
+
+engine = FabricSpark(
+    lakehouse_name="lakehouse",
+    lakehouse_schema_name="lakebench",
+)
+```
+
+## Configuration
+
+- `lakehouse_name`: Fabric Lakehouse name.
+- `lakehouse_schema_name`: schema used for benchmark tables.
+- `spark_measure_telemetry`: enables Spark Measure stage metrics.
+- `collect_stats_on_write`: enables Fabric Delta extended statistics during writes; defaults to `True`.
+- `cost_per_vcore_hour` or `cost_per_hour`: mutually exclusive manual overrides.
+- `tblproperties`: Delta properties applied at table creation.
+
+`compute_stats_all_cols` is a deprecated alias for `collect_stats_on_write`. Explicit benchmark statistics should use the benchmark's `analyze` option.
+
+## Pricing and metadata
+
+When no manual rate is supplied, LakeBench derives a Fabric vCore rate from visible capacity metadata. Results include the capacity ID, computed hourly cost, Spark history URL, runtime version, and relevant Fabric Spark configuration.
+
+Fast optimize is temporarily disabled while LakeBench runs explicit compaction so the measured operation uses the intended Delta optimization path.
+
+Native TPC `.tbl` and `.dat` input is supported through the shared Spark loader.

docs/engines/hdi-spark/README.md

@@ -0,0 +1,39 @@
+# HDInsight Spark
+
+`HDISpark` adapts the generic Spark engine for Azure HDInsight Spark clusters.
+
+## Support
+
+- Benchmarks: ELTBench, TPC-DS, TPC-H, and ClickBench.
+- Runtime: HDInsight Spark.
+- Storage: storage configured for the HDInsight cluster.
+- Authentication: cluster/runtime credentials and Hadoop configuration.
+
+## Installation
+
+HDInsight supplies PySpark. Install LakeBench on the cluster or notebook:
+
+```python
+%pip install lakebench
+```
+
+## Example
+
+```python
+from lakebench.engines import HDISpark
+
+engine = HDISpark(schema_name="lakebench")
+```
+
+## Configuration
+
+- `schema_name`: benchmark schema/database.
+- `spark_measure_telemetry`: optional Spark Measure stage metrics.
+- `cost_per_vcore_hour` or `cost_per_hour`: mutually exclusive manual pricing inputs.
+- `tblproperties`: Delta table properties.
+
+## Pricing
+
+HDInsight does not automatically assemble cluster pricing. Supply an all-inclusive `cost_per_hour` or a `cost_per_vcore_hour` appropriate for the cluster when estimated job cost is required.
+
+Native TPC `.tbl` and `.dat` input is supported through the shared Spark loader.