Skip to content

Commit 7cec97c

Browse files
committed
examples: make the examples a module of their own, and run Spanner Omni in CI
The examples' drivers were becoming dependencies of sqlc itself, and the DuckDB driver, being cgo, had to live in a module apart. Move every example into one module under examples/, which requires sqlc through a replace, so the drivers are its dependencies alone and cgo can stay on for all of it; sqlc's own go.mod loses them, and lib/pq with them, since only the examples imported it. The DuckDB example rejoins the authors example beside the others, and the test helpers for ClickHouse, SQL Server and Spanner move into the examples module, since the drivers they need are what kept them out of sqlc. The end-to-end suite skips the directory under examples that holds those helpers rather than treating it as an example. sqlc-test-setup now installs Spanner Omni from its standalone server release, checked against a pinned SHA-256, and starts a single server in the background on port 15000, so CI runs the Spanner example against it alongside the ClickHouse and SQL Server services. The Makefile and CI run the examples from their module. Drop the clearing of a column's type expression in combineAnalysis: the compiler builds a live-database analyzer only on the legacy path and returns before doing so on the core path, so the two never meet and the expression is never set where the analyzer runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015rfiHyC4iRLyRWc3UmYEtT
1 parent 298b452 commit 7cec97c

31 files changed

Lines changed: 729 additions & 398 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,8 @@ jobs:
6767
run: go build ./...
6868
working-directory: internal/endtoend/testdata
6969

70+
# Installs and starts PostgreSQL, MySQL and Spanner Omni; ClickHouse and
71+
# SQL Server run as services above.
7072
- name: install databases
7173
run: go run ./cmd/sqlc-test-setup install
7274

@@ -83,13 +85,20 @@ jobs:
8385
MYSQL_SERVER_URI: "root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatements=true&parseTime=true"
8486
CLICKHOUSE_SERVER_URI: "clickhouse://default:mysecretpassword@127.0.0.1:9000"
8587
MSSQL_SERVER_URI: "sqlserver://sa:Mysecretpassword1!@127.0.0.1:1433?encrypt=disable"
88+
SPANNER_SERVER_URI: "localhost:15000"
8689
CGO_ENABLED: "0"
8790

88-
# The DuckDB example is its own module: its driver is cgo, which the
89-
# main module's builds keep off.
90-
- name: test examples/duckdb
91+
# The examples are a module of their own, so their drivers are not
92+
# dependencies of sqlc, and cgo stays on for the DuckDB driver.
93+
- name: test examples
9194
run: go test --tags=examples ./...
92-
working-directory: examples/duckdb
95+
working-directory: examples
96+
env:
97+
POSTGRESQL_SERVER_URI: "postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable"
98+
MYSQL_SERVER_URI: "root:mysecretpassword@tcp(127.0.0.1:3306)/mysql?multiStatements=true&parseTime=true"
99+
CLICKHOUSE_SERVER_URI: "clickhouse://default:mysecretpassword@127.0.0.1:9000"
100+
MSSQL_SERVER_URI: "sqlserver://sa:Mysecretpassword1!@127.0.0.1:1433?encrypt=disable"
101+
SPANNER_SERVER_URI: "localhost:15000"
93102

94103
vuln_check:
95104
runs-on: ubuntu-24.04

‎CLAUDE.md‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ This document provides essential information for working with the sqlc codebase,
1212

1313
## Database Setup with sqlc-test-setup
1414

15-
The `sqlc-test-setup` tool (`cmd/sqlc-test-setup/`) automates installing and starting PostgreSQL and MySQL for tests. Both commands are idempotent and safe to re-run.
15+
The `sqlc-test-setup` tool (`cmd/sqlc-test-setup/`) automates installing and starting PostgreSQL, MySQL and Spanner Omni for tests. Both commands are idempotent and safe to re-run.
1616

1717
### Install databases
1818

@@ -24,6 +24,7 @@ This will:
2424
- Configure the apt proxy (if `http_proxy` is set, e.g. in Claude Code remote environments)
2525
- Install PostgreSQL via apt
2626
- Download and install MySQL 26.7 from Oracle's deb bundle
27+
- Download the Spanner Omni standalone server release into the user cache (linux/amd64 only)
2728
- Resolve all dependencies automatically
2829
- Skip anything already installed
2930

@@ -36,12 +37,16 @@ go run ./cmd/sqlc-test-setup start
3637
This will:
3738
- Start PostgreSQL and configure password auth (`postgres`/`postgres`)
3839
- Start MySQL via `mysqld_safe` and set root password (`mysecretpassword`)
39-
- Verify both connections
40+
- Start a Spanner Omni single server in the background, serving plaintext gRPC on port 15000
41+
- Verify the connections
4042
- Skip steps that are already done (running services, existing config)
4143

4244
Connection URIs after start:
4345
- PostgreSQL: `postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable`
4446
- MySQL: `root:mysecretpassword@tcp(127.0.0.1:3306)/mysql`
47+
- Spanner Omni: `localhost:15000` (`SPANNER_SERVER_URI`)
48+
49+
ClickHouse and SQL Server are not installed by the tool; `docker-compose.yml` runs them locally and CI runs them as services. Their tests read `CLICKHOUSE_SERVER_URI` and `MSSQL_SERVER_URI` and skip when unset.
4550

4651
### Run tests
4752

@@ -172,9 +177,13 @@ them as an artifact.
172177

173178
### Example Tests
174179

175-
- **Location:** `/examples/` directory
176-
- **Requirements:** Tagged with "examples", requires live databases
177-
- **Databases:** PostgreSQL, MySQL, SQLite examples
180+
- **Location:** `/examples/` directory, a Go module of its own so the
181+
drivers the examples run against are not dependencies of sqlc, and so
182+
cgo stays on for the DuckDB driver
183+
- **Requirements:** Tagged with "examples", requires live databases; run
184+
with `cd examples && go test --tags=examples ./...`
185+
- **Databases:** PostgreSQL, MySQL, SQLite, ClickHouse, DuckDB, Spanner and
186+
SQL Server examples; each skips when its server is not named
178187

179188
## Database Services
180189

‎Makefile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,11 @@ test-managed:
1414

1515
vet:
1616
go vet ./...
17+
cd ./examples && go vet --tags=examples ./...
1718

1819
test-examples:
1920
go test --tags=examples ./...
20-
cd ./examples/duckdb && go test --tags=examples ./...
21+
cd ./examples && go test --tags=examples ./...
2122

2223
build-endtoend:
2324
cd ./internal/endtoend/testdata && go build ./...
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
//go:build !unix
2+
3+
package main
4+
5+
import "syscall"
6+
7+
func detachedProcess() *syscall.SysProcAttr {
8+
return nil
9+
}

‎cmd/sqlc-test-setup/detach_unix.go‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
//go:build unix
2+
3+
package main
4+
5+
import "syscall"
6+
7+
// detachedProcess puts a child in a session of its own, so it outlives
8+
// this process and is not stopped by the signals this process gets.
9+
func detachedProcess() *syscall.SysProcAttr {
10+
return &syscall.SysProcAttr{Setsid: true}
11+
}

‎cmd/sqlc-test-setup/main.go‎

Lines changed: 184 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"fmt"
77
"io"
88
"log"
9+
"net"
910
"net/http"
1011
"os"
1112
"os/exec"
@@ -19,8 +20,32 @@ import (
1920
const (
2021
// pgVersion is the PostgreSQL version to install.
2122
pgVersion = "18.2.0"
23+
24+
// omniVersion is the Spanner Omni release to install, from its
25+
// standalone server binaries.
26+
omniVersion = "2026.r2.1-beta"
27+
28+
// omniPort is the gRPC port a Spanner Omni single server listens on.
29+
omniPort = "15000"
2230
)
2331

32+
// omniBinary contains the download information for a Spanner Omni server
33+
// release, published at https://storage.googleapis.com/spanner-omni/ and
34+
// documented at https://docs.cloud.google.com/spanner-omni/download.
35+
type omniBinary struct {
36+
URL string
37+
SHA256 string
38+
}
39+
40+
// omniBinaries maps "<GOOS>/<GOARCH>" to the server download. Only linux
41+
// x86_64 is published; the other platforms skip Spanner Omni.
42+
var omniBinaries = map[string]omniBinary{
43+
"linux/amd64": {
44+
URL: "https://storage.googleapis.com/spanner-omni/" + omniVersion + "/spanner-omni-server-" + omniVersion + "-linux-x86_64.tar.gz",
45+
SHA256: "792ffc772d5fff8ade56a8f336993d671d22794fe9f7e88e38a6609c65e16d90",
46+
},
47+
}
48+
2449
// pgBinary contains the download information for a PostgreSQL binary release.
2550
type pgBinary struct {
2651
URL string
@@ -158,6 +183,10 @@ func runInstall() error {
158183
return fmt.Errorf("installing mysql: %w", err)
159184
}
160185

186+
if err := installSpannerOmni(); err != nil {
187+
return fmt.Errorf("installing spanner omni: %w", err)
188+
}
189+
161190
log.Println("=== Install complete ===")
162191
return nil
163192
}
@@ -407,12 +436,164 @@ func runStart() error {
407436
return fmt.Errorf("starting mysql: %w", err)
408437
}
409438

410-
log.Println("=== Both databases are running and configured ===")
411-
log.Println("PostgreSQL: postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable")
412-
log.Println("MySQL: root:mysecretpassword@tcp(127.0.0.1:3306)/mysql")
439+
if err := startSpannerOmni(); err != nil {
440+
return fmt.Errorf("starting spanner omni: %w", err)
441+
}
442+
443+
log.Println("=== Databases are running and configured ===")
444+
log.Println("PostgreSQL: postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable")
445+
log.Println("MySQL: root:mysecretpassword@tcp(127.0.0.1:3306)/mysql")
446+
log.Println("Spanner Omni: localhost:" + omniPort)
413447
return nil
414448
}
415449

450+
// omniDir is where the Spanner Omni release is unpacked: the bin directory
451+
// holds the spanner launcher and spanner_server, and data holds what a
452+
// started server writes.
453+
func omniDir() (string, error) {
454+
cache, err := os.UserCacheDir()
455+
if err != nil {
456+
return "", err
457+
}
458+
return filepath.Join(cache, "sqlc-spanner-omni", omniVersion), nil
459+
}
460+
461+
func omniLauncher(dir string) string {
462+
return filepath.Join(dir, "google", "spanner", "bin", "spanner")
463+
}
464+
465+
// installSpannerOmni downloads the Spanner Omni server release into the
466+
// cache and unpacks it, checking the download against the pinned SHA-256.
467+
// It is a no-op when the release is already unpacked, and skips platforms
468+
// the server is not published for.
469+
func installSpannerOmni() error {
470+
log.Printf("--- Installing Spanner Omni %s ---", omniVersion)
471+
472+
platform := runtime.GOOS + "/" + runtime.GOARCH
473+
bin, ok := omniBinaries[platform]
474+
if !ok {
475+
log.Printf("spanner omni is not published for %s, skipping", platform)
476+
return nil
477+
}
478+
479+
dir, err := omniDir()
480+
if err != nil {
481+
return err
482+
}
483+
if _, err := os.Stat(omniLauncher(dir)); err == nil {
484+
log.Printf("spanner omni %s is already installed in %s", omniVersion, dir)
485+
return nil
486+
}
487+
if err := os.MkdirAll(dir, 0o755); err != nil {
488+
return err
489+
}
490+
491+
archive := filepath.Join(dir, "server.tar.gz")
492+
log.Printf("downloading %s", bin.URL)
493+
if err := downloadFile(archive, bin.URL); err != nil {
494+
return fmt.Errorf("downloading spanner omni: %w", err)
495+
}
496+
defer os.Remove(archive)
497+
498+
sum, err := sha256File(archive)
499+
if err != nil {
500+
return err
501+
}
502+
if sum != bin.SHA256 {
503+
return fmt.Errorf("spanner omni download has SHA-256 %s, want %s", sum, bin.SHA256)
504+
}
505+
506+
log.Printf("unpacking into %s", dir)
507+
if err := run("tar", "-xzf", archive, "-C", dir); err != nil {
508+
return fmt.Errorf("unpacking spanner omni: %w", err)
509+
}
510+
if _, err := os.Stat(omniLauncher(dir)); err != nil {
511+
return fmt.Errorf("spanner omni release does not hold %s", omniLauncher(dir))
512+
}
513+
return nil
514+
}
515+
516+
// startSpannerOmni starts a Spanner Omni single server in the background,
517+
// serving plaintext gRPC on omniPort, and waits until it accepts
518+
// connections. It is a no-op when a server is already listening, and skips
519+
// platforms the server is not installed on.
520+
func startSpannerOmni() error {
521+
log.Println("--- Starting Spanner Omni ---")
522+
523+
if omniReady() {
524+
log.Println("spanner omni is already running and accepting connections")
525+
return nil
526+
}
527+
528+
dir, err := omniDir()
529+
if err != nil {
530+
return err
531+
}
532+
launcher := omniLauncher(dir)
533+
if _, err := os.Stat(launcher); err != nil {
534+
if _, ok := omniBinaries[runtime.GOOS+"/"+runtime.GOARCH]; !ok {
535+
log.Printf("spanner omni is not published for %s/%s, skipping", runtime.GOOS, runtime.GOARCH)
536+
return nil
537+
}
538+
return fmt.Errorf("spanner omni is not installed: run `sqlc-test-setup install` first")
539+
}
540+
541+
data := filepath.Join(dir, "data")
542+
if err := os.MkdirAll(data, 0o755); err != nil {
543+
return err
544+
}
545+
logFile, err := os.Create(filepath.Join(dir, "server.log"))
546+
if err != nil {
547+
return err
548+
}
549+
defer logFile.Close()
550+
551+
// The launcher supervises the server processes for as long as it runs,
552+
// so it is detached from this process and left running.
553+
cmd := exec.Command(launcher, "start-single-server", "--base-dir", data)
554+
cmd.Stdout = logFile
555+
cmd.Stderr = logFile
556+
cmd.SysProcAttr = detachedProcess()
557+
log.Printf("starting %s start-single-server --base-dir %s", launcher, data)
558+
if err := cmd.Start(); err != nil {
559+
return fmt.Errorf("starting spanner omni: %w", err)
560+
}
561+
if err := cmd.Process.Release(); err != nil {
562+
return err
563+
}
564+
565+
log.Println("waiting for spanner omni to accept connections")
566+
if err := waitForSpannerOmni(3 * time.Minute); err != nil {
567+
return fmt.Errorf("spanner omni did not start in time (see %s): %w", logFile.Name(), err)
568+
}
569+
log.Println("spanner omni is accepting connections")
570+
return nil
571+
}
572+
573+
// omniReady reports whether something accepts connections on the Spanner
574+
// Omni gRPC port.
575+
func omniReady() bool {
576+
conn, err := net.DialTimeout("tcp", "127.0.0.1:"+omniPort, time.Second)
577+
if err != nil {
578+
return false
579+
}
580+
conn.Close()
581+
return true
582+
}
583+
584+
// waitForSpannerOmni polls until the server accepts connections or the
585+
// timeout expires.
586+
func waitForSpannerOmni(timeout time.Duration) error {
587+
deadline := time.Now().Add(timeout)
588+
for time.Now().Before(deadline) {
589+
if omniReady() {
590+
return nil
591+
}
592+
time.Sleep(time.Second)
593+
}
594+
return fmt.Errorf("timed out after %s waiting for spanner omni", timeout)
595+
}
596+
416597
func startPostgreSQL() error {
417598
log.Println("--- Starting PostgreSQL ---")
418599

‎docs/guides/development.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,10 @@ go build -o ~/go/bin/sqlc-gen-json ./cmd/sqlc-gen-json
2020
go test ./...
2121
```
2222

23-
To run the tests in the examples folder, use the `examples` tag.
23+
To run the tests in the examples folder, which is a module of its own, use the `examples` tag from inside it.
2424

2525
```
26-
go test --tags=examples ./...
26+
cd examples && go test --tags=examples ./...
2727
```
2828

2929
These tests require locally-running database instances. Run these databases

‎examples/authors/clickhouse/db_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import (
99

1010
_ "github.com/ClickHouse/clickhouse-go/v2"
1111

12-
"github.com/sqlc-dev/sqlc/internal/sqltest/local"
12+
"github.com/sqlc-dev/sqlc/examples/internal/local"
1313
)
1414

1515
func TestAuthors(t *testing.T) {

0 commit comments

Comments
 (0)