Skip to content

docs(duckdb): add public dataset tutorials - #5512

Merged
drrtuy merged 2 commits into
MariaDB:11.4from
mariadb-AlekseiBukhalov:docs/duckdb-public-datasets
Aug 7, 2026
Merged

docs(duckdb): add public dataset tutorials#5512
drrtuy merged 2 commits into
MariaDB:11.4from
mariadb-AlekseiBukhalov:docs/duckdb-public-datasets

Conversation

@mariadb-AlekseiBukhalov

@mariadb-AlekseiBukhalov mariadb-AlekseiBukhalov commented Aug 7, 2026

Copy link
Copy Markdown

What

Two step-by-step tutorials for the DuckDB storage engine, plus the supporting doc updates:

  • NYC Taxi Trips — load one month of TLC Yellow Taxi Parquet with run_in_duckdb() + read_parquet(), keep the taxi zone lookup in ENGINE=InnoDB, then join DuckDB and InnoDB in a plain SELECT. Follows the structure of the ClickHouse taxi tutorial so the two are comparable.
  • OWID CO₂ Emissions — load a commit-pinned CSV revision, run aggregations and a LAG() window function.
  • docs/mariadb-duckdb-incompatibilities.md — documents TIMESTAMPDIFF() under a new "Not translated by the pushdown layer" heading.
  • README.md — new "Tutorials" section linking both.

Docs only: 4 files, 602 insertions, 0 deletions, no code touched.

Testing

The tutorials are treated as a spec: every SQL block was extracted programmatically from the
markdown, executed against a live server, and every expected-output block was compared cell by
cell
with the actual result.

Environment:

Server 11.4.13-MariaDB, built from this branch (RelWithDebInfo)
Base 11.4 @ 90bea36bfad
Plugin ha_duckdb.so, SHOW ENGINESDUCKDB / YES, maturity gamma
DuckDB submodule v1.5.2
Settings duckdb_allow_run_in_duckdb=ON, local_infile=ON

Data sets (verified by sha256 against the URLs in the tutorials):

File sha256
owid-co2-data.csv 7f78e2b218ce4bb8c538bbec04fdc9a7982e8d40bff972e650df603899edd5f6
yellow_tripdata_2024-01.parquet c4d59da7bbc8abaeeeb1727947ee93d9891a71acb42854bd80db1571b2030510
taxi_zone_lookup.csv 1a99e105092230f8620f301edcca7f80d3080642ff404d28ed957d3fa222c8ed

Result — 13 output blocks compared, all matched:

--- OWID (50,411 rows loaded) ---
OK  COUNT(*) + MIN/MAX/COUNT(DISTINCT)
OK  top-10 emitters
OK  decade aggregation
OK  LAG() China year-over-year

--- NYC (2,964,624 trips + 265 zones) ---
OK  COUNT(*) total_trips          OK  duration buckets
OK  MIN/MAX pickup_datetime       OK  zones count + information_schema engines
OK  AVG(tip_amount)               OK  cross-engine join, airport trips by borough
OK  avg cost by passenger_count   OK  pickup zones ranking
OK  busiest pickup hours

Also checked:

  • Cross-engine pushdown: with duckdb_log_options='DUCKDB_QUERY' the error log shows
    DuckDB: cross-engine pushdown with 1 external table(s) and the pushed-down statement, as the
    tutorial claims.
  • LOAD DATA LOCAL INFILE into InnoDB, and information_schema.TABLES reporting
    taxi_zones / InnoDB and trips / DUCKDB.
  • The tutorial's claim about out-of-range timestamps is measured, not guessed: 18 rows fall outside
    January 2024, and AVG(tip_amount) is 3.34 with and without the month filter.
  • DROP DATABASE cleanup for both tutorials.
  • Every relative link in the new files resolves.

@mariadb-AlekseiBukhalov
mariadb-AlekseiBukhalov force-pushed the docs/duckdb-public-datasets branch from bf1a896 to 826eacf Compare August 7, 2026 17:18
@CLAassistant

CLAassistant commented Aug 7, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@mariadb-AlekseiBukhalov
mariadb-AlekseiBukhalov marked this pull request as ready for review August 7, 2026 17:19
Add two step-by-step tutorials:

- NYC Taxi Trips: load TLC Parquet via run_in_duckdb() + read_parquet(),
  keep the zone lookup in InnoDB, then join the two engines in a plain
  SELECT. Follows the structure of the ClickHouse taxi tutorial.

- OWID CO2 Emissions: load a commit-pinned CSV revision, run
  aggregations and a LAG() window function.

Both tutorials point at security-model.md before telling the reader to
enable duckdb_allow_run_in_duckdb, and both note that ordinary
INSERT/UPDATE/DELETE also work on ENGINE=DuckDB tables so that
run_in_duckdb() does not look like the only write path.

Also documents TIMESTAMPDIFF in mariadb-duckdb-incompatibilities.md and
adds a tutorials section to the README.

Every SQL block and every expected output was extracted from the
markdown, executed, and compared cell by cell on MariaDB 11.4.13 built
from this branch; both data sets are pinned so results stay
reproducible.
@mariadb-AlekseiBukhalov
mariadb-AlekseiBukhalov force-pushed the docs/duckdb-public-datasets branch from 826eacf to 5f75f06 Compare August 7, 2026 17:36
@drrtuy
drrtuy self-requested a review August 7, 2026 20:20

@drrtuy drrtuy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

untested ACK

@drrtuy
drrtuy merged commit 16ddaf8 into MariaDB:11.4 Aug 7, 2026
14 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants