Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
458 changes: 245 additions & 213 deletions Cargo.lock

Large diffs are not rendered by default.

848 changes: 495 additions & 353 deletions Cargo.nix

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,11 @@ snafu = "0.9"
strum = { version = "0.28", features = ["derive"] }
tokio = { version = "1.54", features = ["full"] }
tracing = "0.1"
url = "2.5"
yaml_serde = "0.10"

[patch."https://github.com/stackabletech/operator-rs.git"]
# stackable-operator = { git = "https://github.com/stackabletech//operator-rs.git", branch = "main" }
stackable-operator = { git = "https://github.com/stackabletech//operator-rs.git", branch = "fix/versioned-any-changed-value-type" }
# stackable-operator = { path = "../operator-rs/crates/stackable-operator" }

[workspace.lints.clippy]
Expand Down
18 changes: 9 additions & 9 deletions crate-hashes.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion deploy/helm/hive-operator/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ annotations:
url: https://quay.io/repository/stackable/sdp/hive-operator
artifacthub.io/crds: |-
- kind: HiveCluster
version: v1alpha1
version: v1alpha2
name: hiveclusters.hive.stackable.tech
displayName: HiveCluster
description: A Hive cluster stacklet. This resource is managed by the Stackable operator for Apache Hive.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: hive-postgres-s3
Expand All @@ -14,6 +14,7 @@ spec:
credentialsSecretName: hive-credentials
s3:
reference: minio
warehouseDir: s3a://my-bucket/warehouse/
metastore:
roleGroups:
default:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: hive-postgres-s3
Expand All @@ -14,6 +14,7 @@ spec:
credentialsSecretName: hive-credentials
s3:
reference: minio
warehouseDir: s3a://my-bucket/warehouse/
metastore:
roleGroups:
default:
Expand Down
3 changes: 2 additions & 1 deletion docs/modules/hive/pages/getting_started/first_steps.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ A SecretClass for the credentials to the Minio. The credentials were defined in
include::example$getting_started/hive-minio-credentials-secret-class.yaml[]
----

And lastly the actual Apache Hive cluster definition (`hive-postgres-s3.yaml`). Note how it references the `minio` S3 object
And lastly the actual Apache Hive cluster definition (`hive-postgres-s3.yaml`).
Note how it references the `minio` S3Connection and stores the xref:usage-guide/data-storage.adoc#warehouse-directory[warehouse directory] in the `my-bucket` bucket via `warehouseDir`:

[source,yaml]
----
Expand Down
4 changes: 3 additions & 1 deletion docs/modules/hive/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
:hive: https://hive.apache.org
:github: https://github.com/stackabletech/hive-operator/
:crd: {crd-ref}{crd-ref-release}&operator=hive-operator
:crd-hivecluster: {crd-ref}/hive.stackable.tech/v1alpha1/hivecluster{crd-ref-release}
:crd-hivecluster: {crd-ref}/hive.stackable.tech/v1alpha2/hivecluster{crd-ref-release}
:feature-tracker: https://features.stackable.tech/unified

[.link-bar]
Expand Down Expand Up @@ -37,6 +37,8 @@ Every role group is accessible through its own Service, and there is a Service f
The operator creates a xref:concepts:service_discovery.adoc[service discovery ConfigMap] for the Hive metastore instance.
The discovery ConfigMap contains information on how to connect to the HMS.

Besides its Thrift API, the HMS can also serve the xref:usage-guide/iceberg-rest-catalog.adoc[Iceberg REST catalog API] (experimental).

== Dependencies

The Stackable operator for Apache Hive depends on the Stackable xref:commons-operator:index.adoc[commons], xref:secret-operator:index.adoc[secret] and xref:listener-operator:index.adoc[listener] operators.
Expand Down
30 changes: 12 additions & 18 deletions docs/modules/hive/pages/reference/discovery.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@

The Stackable Operator for Apache Hive publishes a discovery https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#configmap-v1-core[ConfigMap], which exposes a client configuration bundle that allows access to the Apache Hive cluster.

The bundle includes an Apache Thrift connection string to access the Hive Metastore service. This string may be used by other operators or tools to configure their products with access to Hive. Access is limited to services within the same Kubernetes cluster.
The bundle includes an Apache Thrift connection string to access the Hive Metastore service. This string may be used by other operators or tools to configure their products with access to Hive.

== Example

Given the following Hive cluster:

[source,yaml,subs="normal,callouts"]
----
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: {clusterName} # <1>
Expand All @@ -29,6 +29,7 @@ spec:
host: postgresql
database: hive
credentialsSecretName: hive-credentials
warehouseDir: s3a://my-bucket/warehouse/
metastore:
roleGroups:
default: # <3>
Expand All @@ -51,27 +52,20 @@ The resulting discovery ConfigMap is `{namespace}/{clusterName}`.

== Contents

=== Internal access

The `{namespace}/{clusterName}` discovery ConfigMap contains the following fields where `{clusterName}` represents the name, `{namespace}` the namespace of the cluster and `{roleGroup}` a role group of the `metastore` role:
The `{namespace}/{clusterName}` discovery ConfigMap contains the following fields where `{clusterName}` represents the name and `{namespace}` the namespace of the cluster:

`HIVE`::
====
Contains the thrift protocol connection string for the Hive metastore service:
Contains the thrift protocol connection string for the Hive metastore service.
The address and port are the ones published by the `{clusterName}-metastore` xref:listener-operator:listener.adoc[Listener], so they depend on the xref:usage-guide/listenerclass.adoc[ListenerClass] in use.
For the default `cluster-internal` ListenerClass this is:

[subs="attributes"]
thrift://{clusterName}.{namespace}.svc.cluster.local:{metastorePort}
====

WARNING: Using the Hive metastore in high availability mode (replicas > 1) does not work with Derby but instead requires a properly configured database like PostgreSQL or MySQL.

=== External access

If `spec.clusterConfig.listenerClass` is set to `external-unstable` an additional ConfigMap is generated to expose external access to the cluster. This discovery ConfigMap is reachable via `{namespace}/{clusterName}-nodeport`.
thrift://{clusterName}-metastore.{namespace}.svc.cluster.local:{metastorePort}

This entry is only present if the Thrift API is enabled (`spec.clusterConfig.thriftApi.enabled`, which is the default).
====
Contains the thrift protocol connection string for the Hive metastore NodePort service:

thrift://<node-ip>:<nodeport>
thrift://<other-node-ip>:<nodeport>
====
The discovery ConfigMap does not contain an entry for the xref:usage-guide/iceberg-rest-catalog.adoc[Iceberg REST catalog] yet.

WARNING: Using the Hive metastore in high availability mode (replicas > 1) does not work with Derby but instead requires a properly configured database like PostgreSQL or MySQL.
26 changes: 26 additions & 0 deletions docs/modules/hive/pages/usage-guide/data-storage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,32 @@ However, as soon as you start storing metadata in the HMS that refers to `s3://`
So if you are storing tables in S3 (or HDFS for that matter), you need to give the HMS access to that filesystem as well.
The Stackable Operator currently supports S3 and HDFS.

[#warehouse-directory]
== Warehouse directory

The warehouse directory is the default location for databases and managed tables that are created without an explicit location.
It is required and configured in the top level `clusterConfig`:

[source,yaml]
----
clusterConfig:
warehouseDir: s3a://my-bucket/warehouse/ # <1>
warehouseExternalDir: s3a://my-bucket/warehouse/external/ # <2>
----
<1> Maps to `hive.metastore.warehouse.dir`.
<2> Optional, maps to `hive.metastore.warehouse.external.dir`.
If it is not set, the operator does not configure it and the Hive metastore uses its internal default.
It is required when using the xref:usage-guide/iceberg-rest-catalog.adoc[Iceberg REST catalog], see that docs for details.

When using HDFS, use an HDFS location instead, e.g. `hdfs://my-hdfs-cluster/warehouse/`.

The location must be on a filesystem the HMS has access to (see <<_s3_support,S3 support>> and <<_apache_hdfs_support,Apache HDFS support>> below).
The HMS only accesses it once databases or tables are created there.

WARNING: Always specify the full URI including the scheme (`s3a://`, `s3://` or `hdfs://`).
A path without a scheme, such as `/warehouse`, is resolved against the default filesystem.
Without an HDFS connection this is the local filesystem of the HMS Pod, which is ephemeral: all data written there is lost when the Pod is restarted.

[s3]
== S3 support

Expand Down
8 changes: 5 additions & 3 deletions docs/modules/hive/pages/usage-guide/database-driver.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ This is done by setting the value of the environment variable `METASTORE_AUX_JAR
[source,yaml]
----
---
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: hive-with-drivers
Expand All @@ -156,6 +156,7 @@ spec:
credentialsSecretName: hive-credentials # <2>
s3:
reference: minio # <3>
warehouseDir: s3a://my-bucket/warehouse/ # <3>
metastore:
roleGroups:
default:
Expand Down Expand Up @@ -186,7 +187,7 @@ stringData:

<1> The database connection details matching those given when deploying the MySQL Helm chart
<2> Hive credentials are retrieved from a Secret
<3> A reference to the file store using S3 (this has been omitted from this article for the sake of brevity, but is described in e.g. the xref:getting_started/first_steps.adoc[] guide)
<3> A reference to the file store using S3 and the xref:usage-guide/data-storage.adoc#warehouse-directory[warehouse directory] in it (this has been omitted from this article for the sake of brevity, but is described in e.g. the xref:getting_started/first_steps.adoc[] guide)
<4> Use `envOverrides` to set the driver path
<5> Use `podOverrides` to mount the driver

Expand Down Expand Up @@ -216,7 +217,7 @@ docker build -f ./Dockerfile -t oci.stackable.tech/sdp/hive:4.2.1-stackable0.0.0
[source, yaml]
----
---
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: hive
Expand All @@ -229,6 +230,7 @@ spec:
...
s3:
...
warehouseDir: ...
metastore:
config:
logging:
Expand Down
15 changes: 11 additions & 4 deletions docs/modules/hive/pages/usage-guide/derby-example.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ It should generally be safe to simply use the latest image version that is avail
.Create a single node Apache Hive Metastore cluster using Derby:
[source,yaml]
----
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: simple-hive-derby
Expand All @@ -19,11 +19,15 @@ spec:
clusterConfig:
metadataDatabase:
derby: {}
warehouseDir: s3a://my-bucket/warehouse/ # <1>
metastore:
roleGroups:
default:
replicas: 1
----
<1> The default location for databases and tables.
The Hive metastore only accesses it once databases or tables are created there, which requires access to the bucket as shown below.
See xref:usage-guide/data-storage.adoc#warehouse-directory[Warehouse directory] for details.

WARNING: You should not use the `Derby` database in production. Derby stores data locally which does not work in high availability setups (multiple replicas) and all data is lost after Pod restarts.

Expand All @@ -46,12 +50,12 @@ In order to upload data to minio we need a port-forward to access the web ui.
----
kubectl port-forward service/minio 9001
----
Then, connect to localhost:9001 and login with the user `minio-access-key` and password `minio-secret-key`. Create a bucket and upload data.
Then, connect to localhost:9001 and login with the user `minio-access-key` and password `minio-secret-key`. Create a bucket named `hive` and upload data.

Deploy the hive cluster:
[source,yaml]
----
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: simple-hive-derby
Expand All @@ -68,6 +72,7 @@ spec:
accessStyle: Path
credentials:
secretClass: simple-hive-s3-secret-class
warehouseDir: s3a://hive/warehouse/
metastore:
roleGroups:
default:
Expand Down Expand Up @@ -121,7 +126,7 @@ helm install hive bitnami/postgresql --version=12.1.5 \
.Create Hive Metastore using a PostgreSQL database
[source,yaml]
----
apiVersion: hive.stackable.tech/v1alpha1
apiVersion: hive.stackable.tech/v1alpha2
kind: HiveCluster
metadata:
name: simple-hive-postgres
Expand All @@ -134,6 +139,7 @@ spec:
host: hive-postgresql.default.svc.cluster.local
database: hive
credentialsSecretName: hive-credentials
warehouseDir: s3a://my-bucket/warehouse/ # <1>
metastore:
roleGroups:
default:
Expand All @@ -148,3 +154,4 @@ stringData:
username: hive
password: hive
----
<1> As in the first example, the Hive metastore needs access to this bucket once databases or tables are created there.
Loading
Loading