Skip to content

Full import and export: object scope, dialect-correct dumps, native dump tools per engine #2618

Description

@datlechin

Problem

Import and export cover tables and views only. Everything else a real dump needs is missing:

  • No routines, triggers, events, sequences, user types, or grants in an export.
  • No per-table WHERE, row limit, or column subset.
  • SQL export has no insert mode (IGNORE / REPLACE / UPSERT), no consistent snapshot, no file splitting, no engine-specific header.
  • Native dump tools exist for PostgreSQL only (pg_dump / pg_restore). No mysqldump, mongodump, sqlite3 .dump, bcp, ClickHouse native, DuckDB EXPORT DATABASE.
  • No XLSX, Parquet, NDJSON, or TSV import. No Parquet, NDJSON, TSV, Markdown, HTML, or COPY export.
  • No saved export profiles, no table-to-table transfer between two connections.
  • export_data over MCP writes its own formats instead of reusing the export plugins, so there are two sources of truth.

Proposed solution

Five phases, in order:

  1. Object scope. Generalize PluginExportTable to an object with a kind (table, view, routine, trigger, event, sequence, type, grant) and add defaulted fetch methods to PluginExportDataSource. Export tree becomes database > object group > object.
  2. Data scope and SQL dialect. Per-table WHERE / LIMIT / column subset. SQL options for insert mode, consistent snapshot, hex blobs, complete inserts, split by size or rows.
  3. Native dump per engine. Generalize PostgresDumpService into a service driven by a descriptor each driver declares (binary names, arguments, environment, progress source). Three models to keep separate: shell-out (MySQL, MongoDB, SQL Server), in-process SQL (SQLite, DuckDB, ClickHouse), and server-side to object store (BigQuery, Snowflake, Oracle Data Pump).
  4. Formats and transfer. Parquet, NDJSON, TSV, Markdown, HTML, COPY on export; XLSX, Parquet, NDJSON, TSV on import. Connect an export data source straight to another connection's import sink for table-to-table transfer with no file.
  5. Profiles and parity. Saved export profiles per connection, error report file for skipped rows, Safe Mode gating on row imports, and export_data reusing the export plugins.

Phase 1 changes PluginKit transfer types, so it needs an additive overload and has to ship with an app release, not ahead of one.

Alternatives considered

Adding formats one at a time without opening the object scope first. That leaves every new format stuck exporting tables and views, and the same work has to be redone later.

Related database type

N/A / General

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions