feat(project): add table row limiter to project dump by shyim · Pull Request #1357 · shopware/shopware-cli

Soner (shyim) · 2026-08-12T05:27:13Z

What changed?

Adds a table row limiter to project dump, so large tables can be reduced to a fixed number of rows while keeping the dump referentially consistent.

New CLI flag (repeatable):

shopware-cli project dump --limit order=100 --limit product=500

New .shopware-project.yml section:

dump:
  limit:
    order:
      rows: 100
      order_by: "order_number DESC" # optional, defaults to created_at DESC

How it works: the dumper walks the foreign-key graph of the cached table schemas and automatically filters every table that references a limited table — directly or transitively — with WHERE ... IN (SELECT ...) conditions, so e.g. limiting order also limits order_line_item, order_delivery, and order_delivery_position to rows belonging to the kept orders.

Details:

Composite primary/foreign keys are supported via tuple IN (Shopware's (id, version_id) pairs).
Nullable foreign keys keep their unrelated rows (order_id IS NULL OR order_id IN (...)).
Tables that are only referenced (like product from order_line_item) are not filtered.
Foreign-key cycles terminate safely by keeping a superset of the attached rows.
Existing where: config entries are combined with the limit conditions ((user where) AND (limit condition)).
A limit on an unknown table logs a warning and is skipped; rows < 1 or a table without a primary key is an error.

Why?

Dumping a production database for local development is painful when a handful of tables (orders, logs) hold millions of rows. A plain LIMIT per table would break foreign keys on import; this feature keeps only rows that belong together, producing small dumps that still import cleanly with FOREIGN_KEY_CHECKS enabled.

How was this tested?

New unit tests in internal/mysqldump/limit_test.go covering: FK cascade over multiple levels, nullable FK handling, custom order_by, where composition, FK cycles, unknown tables, and validation errors.
go test ./... and golangci-lint run pass.

Related issue or discussion

—

🤖 Generated with Claude Code

coderabbitai · 2026-08-12T05:27:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9fc3e388-491b-43f1-9beb-6363818b90d3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

Tomasz Turkowski (tturkowski) · 2026-08-12T09:03:59Z

+
+// keptRowsQuery returns a SELECT yielding the given columns of all rows kept
+// for the given affected table.
+func (r *limitResolver) keptRowsQuery(table string, columns []string) string {


I see a little risk here: what if somebody calls --limit order=100 --limit customer=50? It seems that we'd take the 100 latest orders and the 50 latest customers as two independent queries, with nothing keeping them together - so there's no reason the customers referenced by those 100 orders would be among the 50 kept customers. This would result in orders without customer data. Even with --limit order=100 --limit customer=100, the same problem applies, since we're picking the newest records independently in each case rather than filtering one by the other.

It feels like we could add a condition here that limits the related table's results (as it already does), but also filters them first by the FK condition against it. Wdyt?

I would like to keep this simple as possible. wdym about:

when user limits: order + order_address, we throw an error that limiting order does influence order_address therefore limiting is not allowed?

Yeah that would make sense and we would be fair transparent to the user!

Tomasz Turkowski (@tturkowski) updated can you take a look second time?

Also think keeping it as simple as possible is the way to go here. Applying the limits "top down" and rejecting further limits on references sounds fine.

Just as already hinted in my other comment, it also comes with a caveat. Your PR description mentions:

New CLI flag (repeatable):
shopware-cli project dump --limit order=100 --limit product=500

Details:

Tables that are only referenced (like product from order_line_item) are not filtered.

What does this mean in practice? Can you have that example and supply both a order and product limit? I could imagine you will then fetch more products than the specified limit if the orders also referenced other products? And then the edge case of product variants / self references (topic of my other thread)

Adds a --limit flag (e.g. --limit order=100) and a dump.limit project config section to restrict how many rows of a table are dumped. Tables referencing the limited table via foreign keys, directly or transitively, are filtered automatically so the dump only contains rows belonging to the kept rows. Rows are ordered by created_at DESC by default; the order is configurable via order_by. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The row-limit conditions contain subqueries on other tables (or the table itself via the derived limit table). MySQL rejects those with ER_TABLE_NOT_LOCKED (error 1100) while the table is locked through FLUSH TABLES ... WITH READ LOCK, so locking is now skipped for tables carrying a limit filter and an info log explains why. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Malte Janz (MalteJanz)

Only had a rough first look and didn't had time to review everything in detail yet, but feel free to continue without my full review (don't want to be the blocker while I have limited time) 🙂

Malte Janz (MalteJanz) · 2026-08-13T09:51:17Z

+			if projectCfg.ConfigDump.Limit == nil {
+				projectCfg.ConfigDump.Limit = make(map[string]shop.ConfigDumpLimit)
+			}


nit: if can be moved outside / before the for loop as it doesn't use any iteration specific values

Malte Janz (MalteJanz) · 2026-08-13T10:00:58Z

+			dumper.LimitMap = make(map[string]mysqldump.TableLimit, len(projectCfg.ConfigDump.Limit))
+			for table, limit := range projectCfg.ConfigDump.Limit {
+				dumper.LimitMap[table] = mysqldump.TableLimit{Rows: limit.Rows, OrderBy: limit.OrderBy}
+			}


Just curious: why is the type defined in two places and data copied over here?

shopware-cli/internal/shop/config.go

Lines 255 to 261 in 81b4a89

// ConfigDumpLimit restricts how many rows of a table are exported.

type ConfigDumpLimit struct {

// Maximum amount of rows to export for this table

Rows int `yaml:"rows" jsonschema:"required,minimum=1"`

// SQL ORDER BY clause deciding which rows are kept (e.g. "created_at DESC" for the newest rows). Defaults to "created_at DESC" when the table has a created_at column

OrderBy string `yaml:"order_by,omitempty"`

}

shopware-cli/internal/mysqldump/limit.go

Lines 13 to 24 in 81b4a89

// TableLimit restricts how many rows of a table are dumped. All tables

// referencing the limited table via foreign keys (directly or transitively)

// are filtered automatically so they only contain rows attached to the kept

// rows.

type TableLimit struct {

// Rows is the maximum number of rows to dump.

Rows int

// OrderBy decides which rows are kept (e.g. "`created_at` DESC" for the

// latest rows). When empty and the table has a created_at column,

// "`created_at` DESC" is used.

OrderBy string

}

Malte Janz (MalteJanz) · 2026-08-13T10:10:21Z

+	limits := make(map[string]TableLimit, len(d.LimitMap))
+	for table, limit := range d.LimitMap {


see above, copied again here but this time with the purpose of filtering. Why not deleting entries instead? Is keeping the original data important?

Malte Janz (MalteJanz) · 2026-08-13T10:14:07Z

+	userWhere := d.WhereMap[table]
+	limitWhere := d.limitWhere[table]
+
+	switch {
+	case userWhere != "" && limitWhere != "":
+		return fmt.Sprintf("(%s) AND %s", userWhere, limitWhere)
+	case userWhere != "":
+		return userWhere
+	default:
+		return limitWhere
+	}


At this point why not make the where conditions a slice (behind the map to differentiate between DB tables) and then combine them instead of introducing another variable / field. That way its more flexible for further extension and you don't need such a messy switch case + maintain two maps 😅

Malte Janz (MalteJanz) · 2026-08-13T10:25:16Z

+			query += " WHERE " + where
+		}
+		if limit.OrderBy != "" {
+			query += " ORDER BY " + limit.OrderBy


Likely an edge case, but I'm curious how does this implementation deal with:

you only want to dump 100 orders with their respective dependencies / references including the products

products can have a self reference with their parent_id + parent_version_id for linking to the main / container product from product variants. The good thing it's only one level deep so if you would just import products alone you could order by these fields and import the parent_id = null first, making sure the foreign key data is there as well

is a similar thing applied while there is this limiting logic for orders? e.g. what happens if the order line items only reference some variants, is the parent / container / main product then included as well? Is the parent_id silently dropped to null (making the data incorrect)?

*Didn't had time myself yet to try out the current dump command or this PR ✌️

Malte Janz (MalteJanz) · 2026-08-13T10:31:43Z

+
+// keptRowsQuery returns a SELECT yielding the given columns of all rows kept
+// for the given affected table.
+func (r *limitResolver) keptRowsQuery(table string, columns []string) string {


Also think keeping it as simple as possible is the way to go here. Applying the limits "top down" and rejecting further limits on references sounds fine.

Just as already hinted in my other comment, it also comes with a caveat. Your PR description mentions:

New CLI flag (repeatable):
shopware-cli project dump --limit order=100 --limit product=500

Details:

Tables that are only referenced (like product from order_line_item) are not filtered.

What does this mean in practice? Can you have that example and supply both a order and product limit? I could imagine you will then fetch more products than the specified limit if the orders also referenced other products? And then the edge case of product variants / self references (topic of my other thread)

Soner (shyim) mentioned this pull request Aug 12, 2026

refactor(project): move dump logic into internal packages #1358

Open

Soner (shyim) marked this pull request as ready for review August 12, 2026 06:25

Soner (shyim) requested a review from Tomasz Turkowski (tturkowski) August 12, 2026 07:49

somethings (lasomethingsomething) requested a review from Malte Janz (MalteJanz) August 12, 2026 07:52

somethings (lasomethingsomething) assigned Soner (shyim) Aug 12, 2026

Tomasz Turkowski (tturkowski) reviewed Aug 12, 2026

View reviewed changes

Soner (shyim) requested a review from Tomasz Turkowski (tturkowski) August 12, 2026 10:56

Soner (shyim) and others added 3 commits August 13, 2026 10:52

feat(project): add validation for overlapping table limits in dump

81b4a89

Soner (shyim) force-pushed the feat/dump-table-limit branch from 18e0ef1 to 81b4a89 Compare August 13, 2026 08:52

Malte Janz (MalteJanz) reviewed Aug 13, 2026

View reviewed changes

	// ConfigDumpLimit restricts how many rows of a table are exported.
	type ConfigDumpLimit struct {
	// Maximum amount of rows to export for this table
	Rows int `yaml:"rows" jsonschema:"required,minimum=1"`
	// SQL ORDER BY clause deciding which rows are kept (e.g. "created_at DESC" for the newest rows). Defaults to "created_at DESC" when the table has a created_at column
	OrderBy string `yaml:"order_by,omitempty"`
	}

	// TableLimit restricts how many rows of a table are dumped. All tables
	// referencing the limited table via foreign keys (directly or transitively)
	// are filtered automatically so they only contain rows attached to the kept
	// rows.
	type TableLimit struct {
	// Rows is the maximum number of rows to dump.
	Rows int
	// OrderBy decides which rows are kept (e.g. "`created_at` DESC" for the
	// latest rows). When empty and the table has a created_at column,
	// "`created_at` DESC" is used.
	OrderBy string
	}

		limits := make(map[string]TableLimit, len(d.LimitMap))
		for table, limit := range d.LimitMap {

Uh oh!

Conversation

Soner (shyim) commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Why?

How was this tested?

Related issue or discussion

Uh oh!

coderabbitai Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Malte Janz (MalteJanz) Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Malte Janz (MalteJanz) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Malte Janz (MalteJanz) Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Soner (shyim) commented Aug 12, 2026 •

edited

Loading

coderabbitai Bot commented Aug 12, 2026 •

edited

Loading

Malte Janz (MalteJanz) Aug 13, 2026 •

edited

Loading

Malte Janz (MalteJanz) Aug 13, 2026 •

edited

Loading