Move the analyze checks into goldeneye

goldeneye already generates and checks the dialect seeds and reserved a
place for the analysis checks, so the testcheck module folds into it:
the case loader becomes goldeneye/endtoend, reusing goldeneye's diff,
the ClickHouse analysis joins goldeneye's clickhouse package next to the
dialect generator it shares a binary with, and `check` verifies both the
committed dialect and the analyze cases. `go test ./...` in
internal/goldeneye runs the same checks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6XkyWnx7iJFEb8q3AnYps

Author: claude

CLAUDE.md

@@ -147,8 +147,11 @@ the run early. Run a subset to get past one (`-run 'TestReplay/core/^select'`).
 
 The dialect seeds under `/internal/engine/<engine>/dialect/` are generated
 from a live database by `/internal/goldeneye`, a nested module, and its tests
-verify the committed files against one byte for byte. Engines whose database
-is not available skip.
+verify the committed files against one byte for byte. The same module checks
+the `analyze_*` cases under `/internal/endtoend/testdata/` against what the
+database itself reports for them, so a `fixture.sql` next to a case's schema
+gives the queries rows to run against. Engines whose database is not
+available skip.
 
 ```bash
 cd internal/goldeneye
@@ -240,16 +243,15 @@ MYSQL_SERVER_URI="root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatement
     JSONL read by `/internal/core/seed`; the generated parts come from
     `/internal/goldeneye`
 - `/internal/goldeneye/` - Nested module that generates the dialect seeds
-  under `/internal/engine/<engine>/dialect/` from a live database and checks
-  the committed ones against it, one package per engine; see its README
+  under `/internal/engine/<engine>/dialect/` from a live database, checks
+  the committed ones against it, and checks the analyze cases under
+  `/internal/endtoend/testdata/` against what the database reports, one
+  package per engine; see its README
 - `/internal/core/` - The analysis core: catalog, analyzer and dialect seeds
 - `/internal/compiler/` - Query compilation logic
 - `/internal/codegen/` - Code generation for different languages
 - `/internal/config/` - Configuration file parsing
 - `/internal/endtoend/` - End-to-end tests
-- `/internal/testcheck/` - Nested module that verifies the analyze cases under
-  `/internal/endtoend/testdata/` against a real database, one package per
-  engine; see its README
 - `/internal/sqltest/` - Test database setup (Docker, native, local detection)
 - `/examples/` - Example projects for testing
 

internal/goldeneye/README.md

@@ -87,11 +87,29 @@ the hand-written files alone, and the checks do not look at them.
 - `dialect/` — the record types the files are made of, mirrored from
   `internal/core/seed`, and the helpers that write a generated set of files
   into an engine directory or diff it against what is committed.
+- `endtoend/` — finds the analyze cases and compares an engine's answer with
+  a case's committed output.
 - `postgresql/`, `duckdb/`, `clickhouse/`, `sqlite/` — one package per
-  engine, each exposing `Locate`, `Version` and `Generate`, and a test that
-  runs the check.
+  engine, each exposing `Locate`, `Version` and `Generate`, `Analyze` where
+  the engine has an analysis check, and tests that run the checks.
 - `cmd/goldeneye/` — the command.
 
-The analysis checks — verifying the `analyze_*` cases under
-`internal/endtoend/testdata` against what each database itself reports — are
-meant to live here too, alongside the dialect checks.
+## Analysis checks
+
+`check` also verifies the `analyze_*` cases under `internal/endtoend/testdata`
+against what the database itself reports. A case is an
+`analyze_<name>/<engine>` directory whose `exec.json` runs the analyze
+command; `endtoend/` finds them. The engine package loads the case's
+`schema.sql` and optional `fixture.sql` into the database, runs `query.sql`
+there, prints what the database reports in the JSON shape `sqlc analyze`
+prints, and compares it with the committed `output.json` byte for byte. A
+difference means sqlc's analysis disagrees with the database. A case that
+asks for `--ast` is skipped, since only sqlc can print that.
+
+- **`clickhouse`** runs each case in an ephemeral `clickhouse local` process.
+  Column types come from the executed query's result header, provenance from
+  `EXPLAIN QUERY TREE`, and parameters from sentinel constants substituted for
+  `?`, `sqlc.arg()` and `sqlc.narg()`, since ClickHouse itself never sees a
+  placeholder; `INSERT ... VALUES` parameters map onto `DESCRIBE TABLE`.
+
+The other engines have no analysis check yet.

internal/testcheck/clickhouse/analyze.go internal/goldeneye/clickhouse/analyze.gointernal/testcheck/clickhouse/analyze.go renamed to internal/goldeneye/clickhouse/analyze.go

@@ -1,13 +1,3 @@
-// Package clickhouse verifies the ClickHouse analyze cases under
-// internal/endtoend/testdata against what ClickHouse itself reports.
-//
-// Each case's schema and fixture are loaded into an ephemeral
-// `clickhouse local` process and its queries are run there. Result column
-// types come from the executed query's result header, provenance from
-// EXPLAIN QUERY TREE, and parameters from sentinel constants substituted
-// for the placeholders, since ClickHouse itself never sees a ?. The answer
-// is printed in the JSON shape sqlc analyze prints and compared with the
-// case's committed output.json byte for byte.
 package clickhouse
 
 import (
@@ -17,12 +7,9 @@ import (
 	"fmt"
 	"os"
 
-	"github.com/sqlc-dev/sqlc/internal/testcheck/endtoend"
+	"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
 )
 
-// Engine is the name of this engine's directory under each analyze case.
-const Engine = "clickhouse"
-
 // Analyze runs a case's queries through the clickhouse binary and returns
 // the analysis in the JSON shape sqlc analyze prints.
 func Analyze(ctx context.Context, binary string, c endtoend.Case) ([]byte, error) {

internal/testcheck/clickhouse/check.go internal/goldeneye/clickhouse/check.gointernal/testcheck/clickhouse/check.go renamed to internal/goldeneye/clickhouse/check.go

@@ -9,6 +9,15 @@
 // system.functions carries no signatures — so functions.jsonl is written by
 // hand and is not this package's business.
 //
+// The package also verifies the ClickHouse analyze cases under
+// internal/endtoend/testdata against the same binary: each case's schema and
+// fixture are loaded into a `clickhouse local` process and its queries run
+// there. Result column types come from the executed query's result header,
+// provenance from EXPLAIN QUERY TREE, and parameters from sentinel constants
+// substituted for the placeholders, since ClickHouse itself never sees a ?.
+// The answer is printed in the JSON shape sqlc analyze prints and compared
+// with the case's committed output.json byte for byte.
+//
 // The binary is downloaded once per pinned version by Install, or supplied
 // through the CLICKHOUSE environment variable.
 package clickhouse

internal/goldeneye/clickhouse/clickhouse.go

@@ -5,6 +5,7 @@ import (
 	"testing"
 
 	"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
+	"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
 )
 
 // TestDialect verifies the committed ClickHouse dialect against what the
@@ -35,3 +36,31 @@ func TestDialect(t *testing.T) {
 		t.Errorf("%s does not match what %s reports:\n%s", dir, version, report)
 	}
 }
+
+// TestAnalyzeCases verifies every ClickHouse analyze case under
+// internal/endtoend/testdata against what ClickHouse reports. It skips
+// unless the binary is installed.
+func TestAnalyzeCases(t *testing.T) {
+	binary, err := Locate()
+	if err != nil {
+		t.Skip(err)
+	}
+	cases, err := endtoend.Cases(Engine)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(cases) == 0 {
+		t.Fatal("no clickhouse analyze cases found")
+	}
+	for _, c := range cases {
+		t.Run(c.Name, func(t *testing.T) {
+			diff, err := Check(context.Background(), binary, c)
+			if err != nil {
+				t.Fatal(err)
+			}
+			if diff != "" {
+				t.Errorf("%s does not match what ClickHouse reports (-committed +clickhouse):\n%s", c.Output, diff)
+			}
+		})
+	}
+}

internal/goldeneye/clickhouse/clickhouse_test.go

@@ -1,13 +1,14 @@
 // Command goldeneye generates the dialect seeds under
-// internal/engine/<engine>/dialect from a live database, and checks the
-// committed ones against it.
+// internal/engine/<engine>/dialect from a live database, checks the
+// committed ones against it, and checks the analyze cases under
+// internal/endtoend/testdata against what the database itself reports.
 //
 // Usage, from internal/goldeneye:
 //
 //	go run ./cmd/goldeneye install clickhouse   # download the pinned clickhouse binary
 //	go run ./cmd/goldeneye install sqlite       # build the pinned sqlite3 shells from source
 //	go run ./cmd/goldeneye generate [engine]    # rewrite the generated files from the database
-//	go run ./cmd/goldeneye check [engine]       # compare the committed files with the database
+//	go run ./cmd/goldeneye check [engine]       # compare the committed files and analyze cases with the database
 //
 // Without an engine, generate and check cover every engine whose database
 // is available and say which ones they skipped. `go test ./...` runs the
@@ -22,10 +23,12 @@ import (
 	"io"
 	"os"
 	"runtime"
+	"strings"
 
 	"github.com/sqlc-dev/sqlc/internal/goldeneye/clickhouse"
 	"github.com/sqlc-dev/sqlc/internal/goldeneye/dialect"
 	"github.com/sqlc-dev/sqlc/internal/goldeneye/duckdb"
+	"github.com/sqlc-dev/sqlc/internal/goldeneye/endtoend"
 	"github.com/sqlc-dev/sqlc/internal/goldeneye/postgresql"
 	"github.com/sqlc-dev/sqlc/internal/goldeneye/sqlite"
 )
@@ -44,7 +47,7 @@ const usage = `usage:
   goldeneye generate [engine]
       rewrite the generated dialect files from the database, for every available engine or one
   goldeneye check [engine]
-      compare the committed dialect files with the database, for every available engine or one
+      compare the committed dialect files and analyze cases with the database, for every available engine or one
 
 engines: clickhouse, duckdb, postgresql, sqlite`
 
@@ -58,13 +61,16 @@ type engine struct {
 	version func(context.Context, string) (string, error)
 	// generate reads the dialect from the database.
 	generate func(context.Context, string) (dialect.Files, error)
+	// analyze asks the database what it reports for an analyze case, in
+	// the shape sqlc analyze prints. Nil for an engine without one yet.
+	analyze func(context.Context, string, endtoend.Case) ([]byte, error)
 }
 
 var engines = []engine{
-	{clickhouse.Engine, clickhouse.Locate, clickhouse.Version, clickhouse.Generate},
-	{duckdb.Engine, duckdb.Locate, duckdb.Version, duckdb.Generate},
-	{postgresql.Engine, postgresql.Locate, postgresql.Version, postgresql.Generate},
-	{sqlite.Engine, sqlite.Locate, sqlite.Version, sqlite.Generate},
+	{clickhouse.Engine, clickhouse.Locate, clickhouse.Version, clickhouse.Generate, clickhouse.Analyze},
+	{duckdb.Engine, duckdb.Locate, duckdb.Version, duckdb.Generate, nil},
+	{postgresql.Engine, postgresql.Locate, postgresql.Version, postgresql.Generate, nil},
+	{sqlite.Engine, sqlite.Locate, sqlite.Version, sqlite.Generate, nil},
 }
 
 // installer puts the binary an engine is read through in place, for the
@@ -204,5 +210,37 @@ func check(ctx context.Context, e engine, handle string, stderr io.Writer) error
 		return fmt.Errorf("%s does not match the database\n%s", dir, report)
 	}
 	fmt.Fprintf(stderr, "%s: ok, %d file(s) match\n", e.name, len(files))
+	return checkAnalyzeCases(ctx, e, handle, stderr)
+}
+
+// checkAnalyzeCases compares what the database reports for each of the
+// engine's analyze cases with the case's committed output.
+func checkAnalyzeCases(ctx context.Context, e engine, handle string, stderr io.Writer) error {
+	if e.analyze == nil {
+		return nil
+	}
+	cases, err := endtoend.Cases(e.name)
+	if err != nil {
+		return err
+	}
+	var report strings.Builder
+	for _, c := range cases {
+		got, err := e.analyze(ctx, handle, c)
+		if err != nil {
+			fmt.Fprintf(&report, "%s: %v\n", c.Name, err)
+			continue
+		}
+		diff, err := c.Compare(got)
+		if err != nil {
+			return err
+		}
+		if diff != "" {
+			fmt.Fprintf(&report, "%s (-committed +database)\n%s", c.Name, diff)
+		}
+	}
+	if report.Len() > 0 {
+		return fmt.Errorf("analyze cases do not match the database\n%s", report.String())
+	}
+	fmt.Fprintf(stderr, "%s: ok, %d analyze case(s) match\n", e.name, len(cases))
 	return nil
 }