Repository navigation
Description
Part of stackabletech/issues#889
This is testing that upgrades from the stable release to the nightly release of the operators
and products do not negatively impact the products.
Note
Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:
:green_circle: **airflow-scheduled-job**
The CRD had been updated and I needed to change the following in the manifest:
...The following emojis can be used to indicate the status:
hourglass: In progress / waitingred_circle: In progress with errorsorange_circle: Completed with minor issues or anomaliesgreen_circle: Completed without issues or errors
List of demos
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.
- airflow-scheduled-job @NickLarsenNZ
- argo-cd-git-ops @adwk67
- data-lakehouse-iceberg-trino-spark @sbernauer
- end-to-end-security @Techassi
- hbase-hdfs-load-cycling-data @adwk67
- jupyterhub-keycloak @adwk67
- jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data @adwk67
- logging @Techassi
- nifi-kafka-druid-earthquake-data @razvan
- nifi-kafka-druid-water-level-data @razvan
- opensearch-rag @labrenbe
- signal-processing @Techassi
- spark-k8s-anomaly-detection-taxi-data @adwk67
- trino-iceberg @sbernauer
- trino-taxi-data @sbernauer
List of stacks
Similar to the demos above we need to test all stacks that don't have a demo.
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the stack to indicate a particular stack was picked up for testing.
Tip
Some of the stacks have a tutorial to follow.
- monitoring
- observability
- openldap
- tutorial-openldap @Techassi
Stable to Nightly Upgrade Testing Instructions
These instructions are for deploying and completing the stable demo, and then
upgrading operators, CRDs, and products to the nightly versions well as upgrading
the operators and CRDS.
Tip
Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.
Important
Minio images and charts can no longer be pulled. See @sbernauer's workaround here: release-26.7...2026-11-10-demo-upgrade-testing
Deploy it locally, like:
git checkout 2026-11-10-demo-upgrade-testing
stackablectl demo install airflow-scheduled-job --stack-file stacks/stacks-v2.yaml --demo-file demos/demos-v2.yaml# For demo testing, install demo (stable operators) for the stable release (26.7).
stackablectl demo install <DEMO_NAME>
# For stack testing, install stack via
stackablectl stack install <STACK_NAME>
# --- IMPORTANT ---
# Run through the (still) nightly demo/stack instructions (refer to the list above).
# Install nightly version of operators
stackablectl release upgrade dev
# Bump every stacklet to the version that used by the nightly demo.
# Typically, this is the latest supported version.
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed
# Check that the operator logs look good (no unexpected errors etc.)
# Run through the (still) nightly demo/stack instructions again.HDFS rolling upgrade instructions
For demos that use HDFS, also test a rolling upgrade.
Also see https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/
# In some cases (eg. in the end-to-end-security demo), HDFS is kerberized.
# This requires these commands to be run first.
# Pick "nn/..."
klist -kt /stackable/kerberos/keytab
kinit -kt /stackable/kerberos/keytab <PRINCIPAL_FROM_ABOVE>
# These commands need to be executed in a superuser environment (eg. the namenode pod)
hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query # repeat until "Proceed with rolling upgrade"
# Now patch the productVersion in your regular shell
kubectl patch hdfs/<CLUSTER_NAME> --type=merge \
--patch '{"spec": {"image": {"productVersion": "<NEW_VERSION>"}}}'
# Finalize the upgrade in the namenode pod
# Needs another kinit because the container was restarted
hdfs dfsadmin -rollingUpgrade finalize
kubectl patch hdfs/<CLUSTER_NAME> --subresource=status --type=merge \
--patch '{"status": {"deployedProductVersion": "<NEW_VERSION>"}}'Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
Techassi commentedon Nov 10, 2026
🟠 logging
First, the vector sidecar cannot communicate with the aggregator because of:
This can be fixed by manually upgrading the aggregator to 0.58.0. Fixed on
main.Secondly, the vector aggregator can now not send out the logs it received:
I assume that the vector aggregator is not reading the Opensearch endpoint env var correctly and then fails to lookup the DNS name of the sink. Fixed by fix(logging): Add --dangerously-allow-env-var-interpolation arg #476.
The first two issues are fixed/can be dealt with manually. There is however a third issue after the upgrade. It seems like OpenSearch 3.8.0 doesn't like the index pattern we upload. The "Discover" view fails with
Could not locate that index-pattern-field (id: timestamp). This can also be manually fixed by deleting and re-creating the index pattern manually. What I find most curious is that the timestamp field is named exactly the same... So what's the issue here? @siegfriedweber is looking into it. Fixed by feat(logging): Store log data in OpenSearch data streams #487NickLarsenNZ commentedon Nov 10, 2026
🟢 airflow-scheduled-job
release upgrade changes
product upgrade changes
Find the applicable resources (so far just the ones containing "cluster").
I found the versions based on the stack in
mainat the time:https://github.com/stackabletech/demos/tree/main/stacks/airflow
Techassi commentedon Nov 10, 2026
🟢 signal-processing
sbernauer commentedon Nov 10, 2026
🟢 trino-iceberg Just works
Techassi commentedon Nov 10, 2026
🟢 end-to-end-security
adwk67 commentedon Nov 10, 2026
🟢 jupyterhub-keycloak
selected namespace is not picked up, is usinghas to be installed in default as the change allowing other namespaces was not made until laterdefaultinstead:sbernauer commentedon Nov 11, 2026
🟢 trino-taxi-data just works
adwk67 commentedon Nov 11, 2026
🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data
Product bumps (in addition to op bumps)
(spark: the resource was updated but the pods were not restarted for some reason)
Seems to be fixed with: #480
razvan commentedon Nov 11, 2026
Demos tested
Results
Product bumps (in addition to op bumps)
Pull request
Describes how to flush Druid segments immediately without having to wait an hour: #479
adwk67 commentedon Nov 11, 2026
🟢 spark-k8s-anomaly-detection-taxi-data
Product bumps (in addition to op bumps)
adwk67 commentedon Nov 11, 2026
🟢 hbase-hdfs-load-cycling-data
Product bumps (in addition to op bumps)
Techassi commentedon Nov 12, 2026
🟢 tutorial-openldap (stack)
adwk67 commentedon Nov 12, 2026
🟢 argo-cd-git-ops
Testing this demo against a release is tricky as not only is release-26.7 using minio, but
customGitBranchdefaults to main (demos-v2.yaml:41) as do sealed-secrets.yaml:31 and the DAG git-sync (airflow.yaml:30). UsingcustomGitBranch=release-26.7would not have been sufficient (unless garage was patched into a temporary branch based off of the release) as the names of the init containers changed and they were patched to bump up resources. I started the demo with--parameters customGitBranch=fix/argo-cd-pre-26.11 --stack-parameters stackableReleaseVersion=26.7.0but also had to get Claude to pause ArgoCd and patch various things to get manifest running properly with 26.7. Eventually it was all up and running.Techassi commentedon Nov 13, 2026
Done, closing.