[wayne-x-wang]

Why are the changes needed?

JDBC GetTables only enumerates TableCatalog.listTables for non-session Spark catalogs, omitting persistent views exposed through Spark 4.2's standard ViewCatalog. A VIEW-only request can also return ordinary tables.

Enumerate standard views with their comments and honor JDBC name, namespace, and type filters. Support RelationCatalog, view-only catalogs, and catalogs without SupportsNamespaces by using their default namespace. Discover the standard interface at runtime to retain compatibility with older Spark versions, and preserve the original causes of connector errors. No connector-specific dependency is added.

Include root-namespace views even when a catalog also implements SupportsNamespaces: child namespace enumeration does not cover views stored directly at the root. Probe namespaceExists([]) only for standard ViewCatalog requests whose schema pattern matches the root and whose existing namespace list does not already contain it. List root views only when that probe succeeds; keep table enumeration unchanged. TABLE-only requests, unsupported types, unmatched schema patterns, and the identities-only mode do not add this root probe.

Preserve the existing ignoreTableProperties=true path: ignore type criteria, return table identities with empty comments, and skip persistent-view enumeration and metadata loading.

This replaces the Paimon-specific proposal following review feedback. The original Spark 3.5 / Paimon reproduction in #7772 remains unresolved pending Paimon's migration to the standard API.

How was this patch tested?

Generic JDBC regression tests use in-memory catalog fixtures without Paimon or Iceberg dependencies. The existing Spark 4.2 and 4.3 Maven profiles include the version-specific suite in their CI jobs. The regular Style job checks that source directory even when these profiles are inactive.

Local validation with Java 17:

• Spark 4.2.0 / Scala 2.13: 29 tests passed (9 ordinary V2 catalog tests and 20 ViewCatalog tests). The same 29 tests also passed with the shaded JDBC client and native CI tag exclusions, running the ViewCatalog suite before the ordinary V2 suite. Coverage includes mixed and view-only catalogs, namespace-less catalogs, root views with and without child namespaces, rejection of unsupported root listing, skipped root probes for filtered requests and the fast path, comments and absent comments, type and name filters, connector errors, filtering before metadata loading, and the identities-only fast path.
• Before the root-namespace fix, the 20-test ViewCatalog suite produced 18 passes and 2 failures, with zero suite aborts. Both root-view cases returned no JDBC rows. The same cases pass with the fix.
• Spark 3.5.8 / Scala 2.12: 9 ordinary V2 catalog tests passed.
• dev/reformat completed without source changes; Spotless, Scalastyle, and git diff --check passed. A temporary println violation in the version-specific source was detected under the Spark 3.5 profile; the file was restored byte-for-byte.
• The packaged engine contains SparkViewCatalogUtils and no PaimonCatalogUtils.

Manual integration validation used Spark 4.2.0, a source-built Iceberg Spark 4.2 connector, a local SQLite-backed JDBC catalog with jdbc.schema-version=V1, and the existing Kyuubi 1.10.2 JDBC client. The rebuilt engine was loaded by a separate local Kyuubi 1.13.0-SNAPSHOT instance.

Iceberg sources were checked out at tag apache-iceberg-1.12.0, commit 212aa894e919ff9fdf89160da4aadcbe6858ac01, without source modifications:

./gradlew --no-daemon --max-workers=2 \
  -DsparkVersions=4.2 -DscalaVersion=2.13 \
  :iceberg-spark:iceberg-spark-runtime-4.2_2.13:shadowJar

The tag checkout's default Gradle version calculation produces a JAR labeled 1.13.0-SNAPSHOT when version.txt is absent; the tested code is from the pinned 1.12.0 tag, not Iceberg main or a published Spark 4.1 runtime.

A persistent view was queryable after restarting the instance, before running any view DDL. All 13 JDBC metadata checks passed: TABLE/VIEW separation, null and empty types, lowercase and unsupported types, name/schema patterns, missing comments, the identities-only mode, and a wildcard-schema VIEW request against a connector with no root namespace. For example:

getTables(types=[VIEW])
iceberg | repro7793 | v_empty  | VIEW |
iceberg | repro7793 | v_orders | VIEW | orders view

Run the generic regression suites:

./build/mvn -B -ntp test \
  -pl externals/kyuubi-spark-sql-engine -am \
  -P'spark-4.2,scala-2.13,spark-provided,flink-provided,hive-provided,!jdbc-shaded' \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.engine.spark.operation.SparkCatalogDatabaseOperationSuite,org.apache.kyuubi.engine.spark.operation.ViewCatalogDatabaseOperationSuite

Local Spark 4.3 execution and remote CI for this revision have not been run. The native Spark 4.2/4.3 CI jobs use Java 25; local validation used Java 17.

Was this patch assisted by generative AI tooling?

Assisted-by: Codex (GPT-6)

[github-actions]

Hello @wayne-x-wang,
Thanks for finding the time to submit the Pull request, the Kyuubi community thanks you！
We really appreciate the community's efforts to improve Apache Kyuubi.
If you are new to Apache Kyuubi for code changes, please take some time to read our contributing guide.

[pan3793]

I don't think we should do things like this, Spark introduced ViewCatalog in 4.2. Data source connectors should migrate to this standard API

[wayne-x-wang]

Thanks for the guidance. I agree that Kyuubi should use Spark’s standard ViewCatalog API rather than introduce a Paimon-specific adapter.
I checked Paimon’s latest release, 2.1.0, and its current master. Both still use Paimon’s own SupportView interface, and the published Spark connectors currently go up to Spark 4.1. My original reproduction uses Spark 3.5 and Paimon 2.0, so switching Kyuubi to the standard API would not resolve that case. Upgrading Spark alone would also be insufficient until Paimon adopts the standard interface.
Would it make sense to rework this PR into generic GetTables support for Spark 4.2’s ViewCatalog, removing the Paimon-specific adapter and test setup? I would update the title, description, and tests to reflect that scope, and leave #7772 unresolved pending Paimon’s migration. Alternatively, would you prefer to defer this work until a supported connector provides a concrete integration case?

[pan3793]

It looks like Iceberg 1.12.0 already implemented the ViewCatalog in their Spark 4.2 connector, but they decided to defer publishing the JAR until the next release. That said, you can implement it now and manually test against Iceberg 1.12.0 and Spark 4.2

[wayne-x-wang]

Thanks for pointing this out. I’ll rework this PR into generic GetTables support for Spark’s standard ViewCatalog, removing the Paimon-specific adapter and test setup.
I’ll build the Spark 4.2 connector from the Iceberg 1.12.0 sources and manually verify JDBC table/view metadata against Spark 4.2, alongside generic regression tests. I’ll document the exact source revision, build steps, and validation results in the updated PR description.
I’ll also update the title and scope to make clear that the original Spark 3.5 / Paimon reproduction remains unresolved pending Paimon’s migration.

[wayne-x-wang]

Reworked in 5f2d5c0 to use Spark’s standard ViewCatalog API, removing the Paimon-specific adapter and test setup.

I built the Spark 4.2 connector from the Iceberg 1.12.0 sources and manually verified JDBC metadata against Spark 4.2. All 13 checks passed, and the persistent view remained queryable after restarting Kyuubi. The exact source revision, build steps, and results are documented in the PR description.

Generic JDBC regression tests were added, and all 35 CI checks now pass, including the native Spark 4.2/4.3 jobs on Java 25.

The original Spark 3.5/Paimon reproduction in #7772 remains outside this PR’s scope.

@pan3793 , could you take another look when you have time?