feat: support schema evolution DDLs (ADD/DROP/RENAME/ALTER COLUMN)

[puchengy]

Closes #62.

Adds support for the schema-evolution DDLs requested in #62 by extending BaseLanceNamespaceSparkCatalog.alterTable to handle Spark TableChange.ColumnChange requests, delegating to the underlying Lance dataset schema-evolution API via a new LanceSchemaEvolution helper.

Supported operations

DDL	Lance API
ALTER TABLE ... ADD COLUMN[S]	Dataset.addColumns
ALTER TABLE ... DROP COLUMN [IF EXISTS]	Dataset.dropColumns
ALTER TABLE ... RENAME COLUMN	Dataset.alterColumns (rename)
ALTER TABLE ... ALTER COLUMN ... DROP/SET NOT NULL	Dataset.alterColumns (nullability)

Atomic, all-or-nothing semantics

To honor Spark's alterTable contract, an accepted request is validated in full against the current schema and then committed through exactly one Lance core operation (addColumns / dropColumns / alterColumns — each commits its whole batch atomically):

• Same-kind changes are batched into that single call (e.g. ADD COLUMNS (a, b); rename + nullability edits to the same column are merged into one ColumnAlteration).
• Requests that would require more than one core mutation are rejected before the first write: mixing column additions, drops, and alterations in one statement, or combining any column change with TBLPROPERTIES changes (which commit separately). This keeps a rejected ALTER TABLE from leaving the table partially mutated. Heterogeneous batching in one commit is not yet supported by the core.

Deliberately unsupported (rejected before any mutation)

• ALTER COLUMN ... TYPE — throws UnsupportedOperationException. The lance-core:10.0.0-rc.3 JNI create_column_alteration reads the cast target from the Java ArrowType.toString() (e.g. "Int(64, true)") and parses it with Rust DataType::from_str, swallowing the parse failure with .ok() — so the cast target is dropped and the type change becomes a silent no-op (the commit lands but the stored type is unchanged; verified for both integer- and float-widening). This is fixed upstream in fix(java): carry alterColumns cast type across FFI via C Data Interface lance#8417 (carries the cast type across FFI via the Arrow C Data Interface), but that fix is not yet in a published org.lance:lance-core artifact — the newest published version (11.0.0-beta.3) predates it, and Lance does not publish nightly/snapshot Java builds. Once a lance release containing #8417 ships, this can be enabled by bumping <lance.version> and replacing the rejection with the castTo path.
• ADD COLUMN with a DEFAULT value — Lance's all-null add cannot backfill a default, so it is rejected rather than silently filling NULL.
• ADD COLUMN on a legacy-format (file_format_version='LEGACY') table — the core all-null add path rejects legacy datasets; rejected up front with a clear message instead of a raw core error.
• Positional (FIRST/AFTER) adds, nested-column adds, and column-comment updates.

Tests

• Java unit tests in SparkLanceNamespaceTestBase (run against both directory and REST namespaces): ADD/DROP/RENAME COLUMN and ALTER COLUMN DROP NOT NULL happy paths; batched multi-column ADD; and rejection cases for mixed-kind requests, column-change-plus-TBLPROPERTIES, DROP COLUMN IF EXISTS on a missing column, DEFAULT values, legacy format, and unsupported type changes.
• PySpark integration tests in TestDDLAlterTableColumns.
• TestSparkDirectoryNamespace full suite passes locally (66/66); make lint clean.

Docs updated in docs/src/operations/ddl/alter-table.md.

🤖 Generated with Claude Code

[lance-gatekeeper]

✅ Gate recommendation: approve.

The missing-name ordering fix now preserves DROP COLUMN IF EXISTS while rejecting only existing backtick-named fields that the current core cannot drop. The complete change keeps accepted schema evolution atomic and faithful to Spark’s ordered request semantics.

[fangbo]

Great implementation. I have a problem to discuss. LanceSchemaEvolution operates directly on Dataset . But LanceNamespace also has some interfaces for schema evolution, such as alterTableAddColumns . Do you think it is necessary to call the relevant interfaces of LanceNamespace?

[puchengy]

Thanks for the comment, let me take a look. Thanks!

[puchengy]

Thanks for the suggestion, I think it is the right call, I will implement that in lance-namespace. Thanks for calling this out!

[puchengy]

Hey @jackye1995, can you chime in here?

I looked into @fangbo's suggestion and it turns out reusing lance-namespace is a bigger change than I expected. Here's the gist of what we'd need first:

1. Wire up the Java LanceNamespace alter-column methods to the Rust impls in the lance repo — the Rust side already does the work, but there's no bridge from Java to it yet.
2. Extend the AddColumnsEntry spec in lance-namespace so a plain ADD COLUMN age INT can be expressed — right now it only supports computed/SQL-expression columns.
3. Wait for lance and lance-namespace to cut releases, since this repo can only pick up the changes once they're published.

So it spans three repos and needs to land in order. There are two ways:

• Option 1: merge this PR as is and have a follow up to make above changes.
• Option 2: do the above changes to get things right in the first place.

Could you weigh in by sharing your preference? Since it touches multiple repos, I'd feel better having a maintainer back the approach before I start if we go by option 2. Thanks!