feat(format): add generic metadata API

[lidavidm]

• Add a generic API for fetching metadata, modeled on ADO.NET's GetSchema
• Define the standard metadata collections
• Revert the current revisions to GetObjects in favor of not changing it

Related:

• format: provide way to escape wildcards in GetObjects and other functions #3997: define a way to escape search pattern arguments
• discussion: ConnectionGetObjects vs ConnectionGetTableSchema #1704 (comment): define a way to provide pagination tokens

For consideration:

• Should filters be (optionally) named arguments? (Pointer to struct, or assuming that every filter is preceded by a pointer to a name string.) The justification: we could more easily implement/define "standard" arguments to handle things like the pagination token above without having to replicate it into every single definition. If we only have positional arguments, it starts to get annoying to handle.
• Should filters be typed? (Pointer to struct containing union)
• Should filters be replaced by just leveraging SetOption(Int,...)? (This would obviate the former two points but is rather clunky, and precludes a hypothetical driver that could handle these operations concurrently)

Closes #4400.

[lidavidm]

Also CC @iconara

I've also been thinking about your request for stateless pagination, and I think it's reasonable enough to define a way to get/pass a pagination token. I think there are enough systems that could use it: Athena, BigQuery, Databricks, Snowflake, Iceberg REST catalog, etc.

[lidavidm]

And also CC @mullinsms, who kicked all this off by listing all the types of metadata we didn't support 🙂

In particular, Curt's suggestion means we can support database-specific metadata (though, I'm still of the opinion that something truly database-specific is probably better handled by the application; if the application has a very specific metadata query it wants to issue I'm not sure how useful it is to build it into the driver).

[lidavidm]

At the risk of overcomplicating things: similar to #3623, it might be nice to have a way to request that certain extra fields be included/omitted, e.g. table properties (#3995), where it may be efficient to fetch the data at the same time as the "standard" fields, but where some (many) clients also may not want the field. Maybe it could optionally take an Arrow schema as input for that.

Similarly, maybe the application wants to opt in to run-length-encoding certain response columns to save memory. (Or is that not really a concern so long as things are properly streamed/paginated?)

[lidavidm]

I think I'm leaning towards having all options/filters be set by SetOption. This is perhaps inconvenient for C/C++ users, but language-level bindings can present higher level APIs, and allows us to express better type safety. This would also be consistent with my suggestion in #4317.

Either way, the current declaration needs to clarify the lifetime of the filter argument anyways (the driver should copy arguments as it is not allowed to assume filters will be valid during the returned record reader's lifetime).

I thought about having a SetOptionStringList. This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts, and I would rather avoid trying to encode strings into a single string (via e.g. JSON). But maybe we can embrace Curt's suggestions and lean on Parquet Variant for encoding these sorts of complex-type arguments. That said, I fear I'm reinventing COM or some sort of intraprocess RPC mechanism...

[paleolimbot]

I think I'm leaning towards having all options/filters be set by SetOption.

This is kind of ugly but I see what you're getting at. GDAL handles this with const char** options where options is NULL terminated list of KEY=arbitrary value, with the downside that everything is serialized. If this is a get/set option on a connection is seems more problematic (on a statement it's more clear that options have a specific scope).

This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts

It's a hack, but some REST APIs use the unit separator (\x1f) or have a user-specified separator and just have "schemas" with potentially multiple parts.

I'm guessing you don't want to go this direction, but I can't help but notice there's a large amount of complexity associated with stuffing these concepts into Arrow arrays that are highly nested and very difficult to parse. I'll throw out that you could do something like

struct AdbcCatalogNode {
  AdbcStatus (*get_property)(struct AdbcCatalogNode* self, const char* what, struct ArrowSchema* out_schema, struct ArrowArray* out_array, AdbcError* err);
  AdbcStatus (*get_child)(struct AdbcCatalogNode* self, const char* what, AdbcError* err);
  void* private_data;
  void* private_driver;
}

AdbcStatus AdbcConnectionGetCatalogs(struct AdbcConnection* connection, const char** options, struct AdbcCatalogNode* out, AdbcError* err);

Not perfect, but maybe lets some of this complexity get pushed onto the driver instead of on the consumer since drivers might have abstractions for some of this already.

[zeroshade]

The filters params are missing from the docs here

[zeroshade]

Did I miss it or does the docs not define the semantics of the filters in terms of prefix, a LIKE pattern, regex, etc.?

[lidavidm]

Er, the point of this is that each collection can define its own filters...see the collection definitions.

[zeroshade]

Why REE instead of dictionary encoded (which is more widely supported among arrow implementations)?

[lidavidm]

I suppose I don't like the potential complexity around having to handle replacement/delta dictionaries (which I think are also not quite fully supported in all cases), and the statefulness that dictionaries bring (REE data is always fully decodable from a single batch).

[lidavidm]

I think I'm leaning towards having all options/filters be set by SetOption.

This is kind of ugly but I see what you're getting at. GDAL handles this with const char** options where options is NULL terminated list of KEY=arbitrary value, with the downside that everything is serialized. If this is a get/set option on a connection is seems more problematic (on a statement it's more clear that options have a specific scope).

I agree that the scoping of the option feels a bit awkward. @paleolimbot do you mean that it's a flat list of alternating [key, value, key, value, key, value]? That could work well enough.

This would be useful if we do want to support Iceberg-style catalogs, as we need a way to pass a list of namespace parts

It's a hack, but some REST APIs use the unit separator (\x1f) or have a user-specified separator and just have "schemas" with potentially multiple parts.

I suppose the separator works but I dislike this sort of in-band encoding...that said we could also figure out some convention to encode multiple variadic arguments into a parameter list (e.g. terminating a variadic argument with a NULL element)

I'm guessing you don't want to go this direction, but I can't help but notice there's a large amount of complexity associated with stuffing these concepts into Arrow arrays that are highly nested and very difficult to parse. I'll throw out that you could do something like

struct AdbcCatalogNode {
  AdbcStatus (*get_property)(struct AdbcCatalogNode* self, const char* what, struct ArrowSchema* out_schema, struct ArrowArray* out_array, AdbcError* err);
  AdbcStatus (*get_child)(struct AdbcCatalogNode* self, const char* what, AdbcError* err);
  void* private_data;
  void* private_driver;
}

AdbcStatus AdbcConnectionGetCatalogs(struct AdbcConnection* connection, const char** options, struct AdbcCatalogNode* out, AdbcError* err);

Not perfect, but maybe lets some of this complexity get pushed onto the driver instead of on the consumer since drivers might have abstractions for some of this already.

The new collection definitions don't nest the data anymore, to hopefully avoid this.

[lidavidm]

Also, I think encoding filter arguments is good enough. At least with current APIs, essentially all filter arguments are strings anyways.

[paleolimbot]

do you mean that it's a flat list of alternating [key, value, key, value, key, value]

Yes, like auto options = {"METADATA_ENCODING=geoarrow", "FOOFY2=something else", nullptr};. No need to do anything if the existing filter specification works.

[lidavidm]

Another thought: what if we just place this as a statement-level function? Even if it's not associated with a statement per se this lets us get access to existing functions like RequestSchema and lets us scope SetOption without "affecting" the entire connection.

[lidavidm]

Drafted that at #4694.

I don't love that API either but I think it is more consistent than this one.

[lidavidm]

I'll close this in favor of the reworked proposal at #4694.