Skip to content

tracking: Test demo upgrades to nightly versions for 26.11.0 #473

Closed

Description

@Techassi

Part of stackabletech/issues#889

This is testing that upgrades from the stable release to the nightly release of the operators
and products do not negatively impact the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

  • hourglass: In progress / waiting
  • red_circle: In progress with errors
  • orange_circle: Completed with minor issues or anomalies
  • green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.

List of stacks

Similar to the demos above we need to test all stacks that don't have a demo.
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the stack to indicate a particular stack was picked up for testing.

Tip

Some of the stacks have a tutorial to follow.

  • monitoring
  • observability
  • openldap
  • tutorial-openldap @Techassi

Stable to Nightly Upgrade Testing Instructions

These instructions are for deploying and completing the stable demo, and then
upgrading operators, CRDs, and products to the nightly versions well as upgrading
the operators and CRDS.

Tip

Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.

Important

Minio images and charts can no longer be pulled. See @sbernauer's workaround here: release-26.7...2026-11-10-demo-upgrade-testing
Deploy it locally, like:

git checkout 2026-11-10-demo-upgrade-testing
stackablectl demo install airflow-scheduled-job --stack-file stacks/stacks-v2.yaml --demo-file demos/demos-v2.yaml
# For demo testing, install demo (stable operators) for the stable release (26.7).
stackablectl demo install <DEMO_NAME>

# For stack testing, install stack via
stackablectl stack install <STACK_NAME>

# --- IMPORTANT ---
# Run through the (still) nightly demo/stack instructions (refer to the list above).

# Install nightly version of operators
stackablectl release upgrade dev

# Bump every stacklet to the version that used by the nightly demo.
# Typically, this is the latest supported version.
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed

# Check that the operator logs look good (no unexpected errors etc.)

# Run through the (still) nightly demo/stack instructions again.
HDFS rolling upgrade instructions

For demos that use HDFS, also test a rolling upgrade.
Also see https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/

# In some cases (eg. in the end-to-end-security demo), HDFS is kerberized.
# This requires these commands to be run first.
# Pick "nn/..."
klist -kt /stackable/kerberos/keytab
kinit -kt /stackable/kerberos/keytab <PRINCIPAL_FROM_ABOVE>

# These commands need to be executed in a superuser environment (eg. the namenode pod)
hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query  # repeat until "Proceed with rolling upgrade"

# Now patch the productVersion in your regular shell
kubectl patch hdfs/<CLUSTER_NAME> --type=merge \
  --patch '{"spec": {"image": {"productVersion": "<NEW_VERSION>"}}}'

# Finalize the upgrade in the namenode pod
# Needs another kinit because the container was restarted
hdfs dfsadmin -rollingUpgrade finalize

kubectl patch hdfs/<CLUSTER_NAME> --subresource=status --type=merge \
  --patch '{"status": {"deployedProductVersion": "<NEW_VERSION>"}}'

Activity

  1. moved this from Next to Development: In Progress in Stackable Engineeringon Nov 10, 2026
  2. Techassi commented on Nov 10, 2026

    @Techassi
    MemberAuthor

    🟠 logging

    • 🟢 Stable from scratch
    • 🟠 Upgrade to nightly
      • First, the vector sidecar cannot communicate with the aggregator because of:

        2026-11-10T12:41:02.317935Z ERROR sink{component_kind="sink" component_id=aggregator component_type=vector}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Request failed: status: Unimplemented, message: "compression scheme `zstd` is not supported", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "grpc-accept-encoding": "gzip,identity", "date": "Tue, 10 Nov 2026 12:41:02 GMT"} }
        

        This can be fixed by manually upgrading the aggregator to 0.58.0. Fixed on main.

      • Secondly, the vector aggregator can now not send out the logs it received:

        2026-11-10T12:44:26.393993Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name or service not known component_kind="sink" component_type="elasticsearch" component_id=opensearch
        

        I assume that the vector aggregator is not reading the Opensearch endpoint env var correctly and then fails to lookup the DNS name of the sink. Fixed by fix(logging): Add --dangerously-allow-env-var-interpolation arg #476.

      • The first two issues are fixed/can be dealt with manually. There is however a third issue after the upgrade. It seems like OpenSearch 3.8.0 doesn't like the index pattern we upload. The "Discover" view fails with Could not locate that index-pattern-field (id: timestamp). This can also be manually fixed by deleting and re-creating the index pattern manually. What I find most curious is that the timestamp field is named exactly the same... So what's the issue here? @siegfriedweber is looking into it. Fixed by feat(logging): Store log data in OpenSearch data streams #487

  3. NickLarsenNZ commented on Nov 10, 2026

    @NickLarsenNZ
    Member

    🟢 airflow-scheduled-job

    • 🟢 Stable from scratch
      • Well minio makes things difficult with the stable demos. Used Sebastians branch and added it to this issue description as a tip.
    • 🟢 Upgrade to nightly
    release upgrade changes
    stackablectl release upgrade dev
    === 18:49:25 ControllerRevision airflow-scheduled-job/hive-iceberg-metastore-default-7bd7f9c6fc  (rev 2, owner StatefulSet/hive-iceberg-metastore-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/hive-iceberg-metastore-default
      ± value change
        - 180e76f7-0280-44be-938d-95b6684f43a2/1285
        + 180e76f7-0280-44be-938d-95b6684f43a2/6571
    
    
    === 18:49:25 ControllerRevision airflow-scheduled-job/hive-iceberg-metastore-default-5897d94c8c  (rev 3, owner StatefulSet/hive-iceberg-metastore-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 4.2.0-stackable26.7.0
        + 4.2.0-stackable0.0.0-dev
    
    data.spec.template.spec.containers.hive.args.0
      ± value change in multiline text (one insert, no deletions)
          echo copying /stackable/mount/config to /stackable/config
          cp -RL /stackable/mount/config/* /stackable/config
          echo copying /stackable/mount/log-config/metastore-log4j2.properties to /stackable/config/metastore-log4j2.properties
          cp -RL /stackable/mount/log-config/metastore-log4j2.properties /stackable/config/metastore-log4j2.properties
          
          [34 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          bin/base --config "/stackable/config" --service schemaTool -dbType "postgres" -initOrUpgradeSchema
          bin/base --config "/stackable/config" --service metastore &
          
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.hive.env
      ⇆ order changed                                                                              - one list entry removed:                    + three list entries added:
        - HADOOP_HEAPSIZE, HADOOP_OPTS, METADATA_DATABASE_USERNAME, METADATA_DATABASE_PASSWORD       - name: CONTAINERDEBUG_LOG_DIRECTORY         - name: FILE_LOG_DIRECTORY
        + HADOOP_HEAPSIZE, HADOOP_OPTS, METADATA_DATABASE_PASSWORD, METADATA_DATABASE_USERNAME       │ value: /stackable/log/containerdebug       │ value: /stackable/log/containerdebug
                                                                                                                                                  - name: FILE_LOG_MAX_FILES
                                                                                                                                                  │ value: 2
                                                                                                                                                  - name: FILE_LOG_ROTATION_PERIOD
                                                                                                                                                  │ value: hourly
    
    data.spec.template.spec.containers.hive.image
      ± value change
        - oci.stackable.tech/sdp/hive:4.2.0-stackable26.7.0
        + oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev
    
    data.spec.template.spec.containers.hive.volumeMounts
      ⇆ order changed
        minio-s3-credentials-s3-credentials   config
        tls-ca-cert                           config-mount
        config                                log
        config-mount                          log-config-mount
        log                                   listener
        log-config-mount                      minio-s3-credentials-s3-credentials
        listener                              tls-ca-cert
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        - minio-s3-credentials-s3-credentials, tls-ca-cert, config, config-mount, log, log-config-mount
        + config, config-mount, log, log-config-mount, minio-s3-credentials-s3-credentials, tls-ca-cert
    
    
    === 18:49:29 ControllerRevision airflow-scheduled-job/airflow-dagprocessor-default-79649696f  (rev 2, owner StatefulSet/airflow-dagprocessor-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/1695
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
    
    
    === 18:49:29 ControllerRevision airflow-scheduled-job/airflow-scheduler-default-c89fc4c8f  (rev 2, owner StatefulSet/airflow-scheduler-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/1695
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
    
    
    === 18:49:30 ControllerRevision airflow-scheduled-job/airflow-triggerer-default-c6c6f6bd8  (rev 2, owner StatefulSet/airflow-triggerer-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/1695
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
    
    
    === 18:49:31 ControllerRevision airflow-scheduled-job/airflow-webserver-default-8658c886  (rev 2, owner StatefulSet/airflow-webserver-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/1695
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
    
    
    === 18:49:31 ControllerRevision airflow-scheduled-job/airflow-scheduler-default-57d7b48b4  (rev 3, owner StatefulSet/airflow-scheduler-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-scheduler-default
      ± value change
        - a23be904-cf45-4d4f-a010-f93951a2d717/1799
        + a23be904-cf45-4d4f-a010-f93951a2d717/6650
    
    
    === 18:49:32 ControllerRevision airflow-scheduled-job/airflow-triggerer-default-6764f5d4bd  (rev 3, owner StatefulSet/airflow-triggerer-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-triggerer-default
      ± value change
        - 9df97e6e-a740-4a9b-b5a4-78ad24ed770e/1885
        + 9df97e6e-a740-4a9b-b5a4-78ad24ed770e/6660
    
    
    === 18:49:33 ControllerRevision airflow-scheduled-job/airflow-webserver-default-c97846b6d  (rev 3, owner StatefulSet/airflow-webserver-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-webserver-default
      ± value change
        - d08176fc-944b-4f04-b359-f0b88ea02ce7/1729
        + d08176fc-944b-4f04-b359-f0b88ea02ce7/6649
    
    
    === 18:49:33 ControllerRevision airflow-scheduled-job/airflow-dagprocessor-default-c7958cf4b  (rev 3, owner StatefulSet/airflow-dagprocessor-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-dagprocessor-default
      ± value change
        - e2b9d3a7-24f4-4436-b59a-6f294d4d3176/1849
        + e2b9d3a7-24f4-4436-b59a-6f294d4d3176/6656
    
    
    === 18:49:34 ControllerRevision airflow-scheduled-job/airflow-scheduler-default-78d88c49c4  (rev 4, owner StatefulSet/airflow-scheduler-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable26.7.0
        + 3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.airflow.args.0
      ± value change in multiline text (three inserts, no deletions)
          cp -RL /stackable/app/config/webserver_config.py /stackable/airflow/webserver_config.py
          
          prepare_signal_handlers()
          {
          
          [25 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          airflow db migrate
        + set +x
        + echo 'Running "airflow users create [...]", which is not shown as it leaks the Airflow admin credentials'
          airflow users create --username "$ADMIN_USERNAME" --firstname "$ADMIN_FIRSTNAME" --lastname "$ADMIN_LASTNAME" --email "$ADMIN_EMAIL" --password "$ADMIN_PASSWORD" --role "Admin"
        + set -x
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          airflow scheduler &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.airflow.env
      - two list entries removed:                  + three list entries added:
        - name: AIRFLOW__WEBSERVER__SECRET_KEY       - name: FILE_LOG_DIRECTORY
        │ valueFrom:                                 │ value: /stackable/log/containerdebug
        │ │ secretKeyRef:                            - name: FILE_LOG_MAX_FILES
        │ │ │ name: airflow-internal-secret          │ value: 2
        │ │ │ key: INTERNAL_SECRET                   - name: FILE_LOG_ROTATION_PERIOD
        - name: CONTAINERDEBUG_LOG_DIRECTORY         │ value: hourly
        │ value: /stackable/log/containerdebug
    
    data.spec.template.spec.containers.airflow.volumeMounts
      ⇆ order changed
        /dags/date_demo.py          /stackable/app/config
        /dags/pyspark_pi.py         /stackable/app/log_config
        /dags/pyspark_pi.yaml       /stackable/log
        /dags/kafka.py              /templates
        /dags/triggerer.py          /dags/date_demo.py
        /dags/dbt.py                /dags/pyspark_pi.py
        /stackable/tls-pem          /dags/pyspark_pi.yaml
        /stackable/app/config       /dags/kafka.py
        /stackable/app/log_config   /dags/triggerer.py
        /stackable/log              /dags/dbt.py
        /templates                  /stackable/tls-pem
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable26.7.0
        + oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        - airflow-dags, tls-pem, config, log, log-config, airflow-executor-pod-template
        + config, log, log-config, airflow-executor-pod-template, airflow-dags, tls-pem
    
    
    === 18:49:34 ControllerRevision airflow-scheduled-job/airflow-webserver-default-575866977f  (rev 4, owner StatefulSet/airflow-webserver-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable26.7.0
        + 3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.affinity.podAffinity.preferredDuringSchedulingIgnoredDuringExecution
      + one list entry added:
        - podAffinityTerm:
        │ │ labelSelector:
        │ │ │ matchLabels:
        │ │ │ │ app.kubernetes.io/component: server
        │ │ │ │ app.kubernetes.io/instance: opa
        │ │ │ │ app.kubernetes.io/name: opa
        │ │ topologyKey: kubernetes.io/hostname
        │ weight: 50
    
    data.spec.template.spec.containers.airflow.args.0
      ± value change in multiline text (one insert, no deletions)
          cp -RL /stackable/app/config/webserver_config.py /stackable/airflow/webserver_config.py
          
          prepare_signal_handlers()
          {
          
          [25 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          airflow api-server &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.airflow.env
      - two list entries removed:                  + three list entries added:
        - name: AIRFLOW__WEBSERVER__SECRET_KEY       - name: FILE_LOG_DIRECTORY
        │ valueFrom:                                 │ value: /stackable/log/containerdebug
        │ │ secretKeyRef:                            - name: FILE_LOG_MAX_FILES
        │ │ │ name: airflow-internal-secret          │ value: 2
        │ │ │ key: INTERNAL_SECRET                   - name: FILE_LOG_ROTATION_PERIOD
        - name: CONTAINERDEBUG_LOG_DIRECTORY         │ value: hourly
        │ value: /stackable/log/containerdebug
    
    data.spec.template.spec.containers.airflow.volumeMounts
      ⇆ order changed
        /dags/date_demo.py          /stackable/app/config
        /dags/pyspark_pi.py         /stackable/app/log_config
        /dags/pyspark_pi.yaml       /stackable/log
        /dags/kafka.py              /templates
        /dags/triggerer.py          /stackable/listener
        /dags/dbt.py                /dags/date_demo.py
        /stackable/tls-pem          /dags/pyspark_pi.py
        /stackable/app/config       /dags/pyspark_pi.yaml
        /stackable/app/log_config   /dags/kafka.py
        /stackable/log              /dags/triggerer.py
        /templates                  /dags/dbt.py
        /stackable/listener         /stackable/tls-pem
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable26.7.0
        + oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        - airflow-dags, tls-pem, config, log, log-config, airflow-executor-pod-template
        + config, log, log-config, airflow-executor-pod-template, airflow-dags, tls-pem
    
    
    === 18:49:35 ControllerRevision airflow-scheduled-job/airflow-dagprocessor-default-698d48f488  (rev 4, owner StatefulSet/airflow-dagprocessor-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable26.7.0
        + 3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.airflow.args.0
      ± value change in multiline text (one insert, no deletions)
          cp -RL /stackable/app/config/webserver_config.py /stackable/airflow/webserver_config.py
          
          prepare_signal_handlers()
          {
          
          [25 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          airflow dag-processor &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.airflow.env
      - two list entries removed:                  + three list entries added:
        - name: AIRFLOW__WEBSERVER__SECRET_KEY       - name: FILE_LOG_DIRECTORY
        │ valueFrom:                                 │ value: /stackable/log/containerdebug
        │ │ secretKeyRef:                            - name: FILE_LOG_MAX_FILES
        │ │ │ name: airflow-internal-secret          │ value: 2
        │ │ │ key: INTERNAL_SECRET                   - name: FILE_LOG_ROTATION_PERIOD
        - name: CONTAINERDEBUG_LOG_DIRECTORY         │ value: hourly
        │ value: /stackable/log/containerdebug
    
    data.spec.template.spec.containers.airflow.volumeMounts
      ⇆ order changed
        /dags/date_demo.py          /stackable/app/config
        /dags/pyspark_pi.py         /stackable/app/log_config
        /dags/pyspark_pi.yaml       /stackable/log
        /dags/kafka.py              /templates
        /dags/triggerer.py          /dags/date_demo.py
        /dags/dbt.py                /dags/pyspark_pi.py
        /stackable/tls-pem          /dags/pyspark_pi.yaml
        /stackable/app/config       /dags/kafka.py
        /stackable/app/log_config   /dags/triggerer.py
        /stackable/log              /dags/dbt.py
        /templates                  /stackable/tls-pem
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable26.7.0
        + oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        - airflow-dags, tls-pem, config, log, log-config, airflow-executor-pod-template
        + config, log, log-config, airflow-executor-pod-template, airflow-dags, tls-pem
    
    
    === 18:49:36 ControllerRevision airflow-scheduled-job/airflow-triggerer-default-cb88b7c7d  (rev 4, owner StatefulSet/airflow-triggerer-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable26.7.0
        + 3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.airflow.args.0
      ± value change in multiline text (one insert, no deletions)
          cp -RL /stackable/app/config/webserver_config.py /stackable/airflow/webserver_config.py
          
          prepare_signal_handlers()
          {
          
          [25 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          airflow triggerer &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.airflow.env
      - two list entries removed:                  + three list entries added:
        - name: AIRFLOW__WEBSERVER__SECRET_KEY       - name: FILE_LOG_DIRECTORY
        │ valueFrom:                                 │ value: /stackable/log/containerdebug
        │ │ secretKeyRef:                            - name: FILE_LOG_MAX_FILES
        │ │ │ name: airflow-internal-secret          │ value: 2
        │ │ │ key: INTERNAL_SECRET                   - name: FILE_LOG_ROTATION_PERIOD
        - name: CONTAINERDEBUG_LOG_DIRECTORY         │ value: hourly
        │ value: /stackable/log/containerdebug
    
    data.spec.template.spec.containers.airflow.volumeMounts
      ⇆ order changed
        /dags/date_demo.py          /stackable/app/config
        /dags/pyspark_pi.py         /stackable/app/log_config
        /dags/pyspark_pi.yaml       /stackable/log
        /dags/kafka.py              /templates
        /dags/triggerer.py          /dags/date_demo.py
        /dags/dbt.py                /dags/pyspark_pi.py
        /stackable/tls-pem          /dags/pyspark_pi.yaml
        /stackable/app/config       /dags/kafka.py
        /stackable/app/log_config   /dags/triggerer.py
        /stackable/log              /dags/dbt.py
        /templates                  /stackable/tls-pem
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable26.7.0
        + oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
    
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        - airflow-dags, tls-pem, config, log, log-config, airflow-executor-pod-template
        + config, log, log-config, airflow-executor-pod-template, airflow-dags, tls-pem
    
    
    === 18:49:41 ControllerRevision airflow-scheduled-job/trino-coordinator-default-6d94df9d5  (rev 2, owner StatefulSet/trino-coordinator-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-coordinator-default
      ± value change
        - 05edb008-9739-4a15-8df2-a5cb276c53d9/1705
        + 05edb008-9739-4a15-8df2-a5cb276c53d9/6987
    
    
    === 18:49:42 ControllerRevision airflow-scheduled-job/trino-worker-default-db7b8d768  (rev 2, owner StatefulSet/trino-worker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-worker-default
      ± value change
        - 4409e7ca-3aa5-4abb-abe7-1e3979d47d7f/1778
        + 4409e7ca-3aa5-4abb-abe7-1e3979d47d7f/6990
    
    
    === 18:49:43 ControllerRevision airflow-scheduled-job/trino-coordinator-default-74c49d9d89  (rev 3, owner StatefulSet/trino-coordinator-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-coordinator-default-catalog
      ± value change
        - 30ee77a9-6807-4339-b40d-e589a05b144f/1712
        + 30ee77a9-6807-4339-b40d-e589a05b144f/6988
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 481-stackable26.7.0
        + 481-stackable0.0.0-dev
    
    data.spec.template.spec.affinity.podAffinity.preferredDuringSchedulingIgnoredDuringExecution
      + one list entry added:
        - podAffinityTerm:
        │ │ labelSelector:
        │ │ │ matchLabels:
        │ │ │ │ app.kubernetes.io/component: server
        │ │ │ │ app.kubernetes.io/instance: opa
        │ │ │ │ app.kubernetes.io/name: opa
        │ │ topologyKey: kubernetes.io/hostname
        │ weight: 50
    
    data.spec.template.spec.containers.trino.args.0
      ± value change in multiline text (two inserts, no deletions)
          echo copying /stackable/config to /stackable/rwconfig
          cp -RL /stackable/config/* /stackable/rwconfig
          set +x
          export CATALOG_ICEBERG_S3_AWS_ACCESS_KEY="$(cat /stackable/secrets/minio-s3-credentials/accessKey)"
          export CATALOG_ICEBERG_S3_AWS_SECRET_KEY="$(cat /stackable/secrets/minio-s3-credentials/secretKey)"
          test -f /stackable/rwconfig/exchange-manager.properties && config-utils template /stackable/rwconfig/exchange-manager.properties
          test -f /stackable/rwconfig/spooling-manager.properties && config-utils template /stackable/rwconfig/spooling-manager.properties
        + test -f /stackable/rwconfig/event-listener.properties && config-utils template /stackable/rwconfig/event-listener.properties
          set -x
          
          prepare_signal_handlers()
          {
          
          [25 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          bin/launcher run --etc-dir=/stackable/rwconfig &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.trino.env
      - one list entry removed:                    + three list entries added:
        - name: CONTAINERDEBUG_LOG_DIRECTORY         - name: FILE_LOG_DIRECTORY
        │ value: /stackable/log/containerdebug       │ value: /stackable/log/containerdebug
                                                     - name: FILE_LOG_MAX_FILES
                                                     │ value: 2
                                                     - name: FILE_LOG_ROTATION_PERIOD
                                                     │ value: hourly
    
    data.spec.template.spec.containers.trino.env.INTERNAL_SECRET.valueFrom.secretKeyRef
      - one map entry removed:
        optional: false
    
    data.spec.template.spec.containers.trino.env.SPOOLING_SECRET.valueFrom.secretKeyRef
      - one map entry removed:
        optional: false
    
    data.spec.template.spec.containers.trino.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable26.7.0
        + oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
    
    data.spec.template.spec.containers.trino.volumeMounts
      ⇆ order changed
        users                                 config
        server-tls-mount                      rwconfig
        server-tls                            catalog
        client-tls                            log
        internal-tls-mount                    listener
        internal-tls                          server-tls-mount
        minio-s3-credentials-s3-credentials   server-tls
        tls-ca-cert                           client-tls
        listener                              internal-tls-mount
        config                                internal-tls
        rwconfig                              users
        catalog                               minio-s3-credentials-s3-credentials
        log                                   tls-ca-cert
    data.spec.template.spec.containers.password-file-updater.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable26.7.0
        + oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
    
    data.spec.template.spec.containers.password-file-updater.volumeMounts
      ⇆ order changed
        - users, trino-users, log
        + log, users, trino-users
    
    data.spec.template.spec.initContainers.prepare.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable26.7.0
        + oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
    
    data.spec.template.spec.initContainers.prepare.volumeMounts
      ⇆ order changed
        server-tls-mount                      rwconfig
        server-tls                            log-config
        client-tls                            log
        internal-tls-mount                    server-tls-mount
        internal-tls                          server-tls
        minio-s3-credentials-s3-credentials   client-tls
        tls-ca-cert                           internal-tls-mount
        rwconfig                              internal-tls
        log-config                            minio-s3-credentials-s3-credentials
        log                                   tls-ca-cert
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        trino-users                           config
        users                                 rwconfig
        server-tls-mount                      catalog
        server-tls                            log-config
        client-tls                            log
        internal-tls-mount                    server-tls-mount
        internal-tls                          server-tls
        minio-s3-credentials-s3-credentials   client-tls
        tls-ca-cert                           internal-tls-mount
        log-config                            internal-tls
        config                                trino-users
        rwconfig                              users
        catalog                               minio-s3-credentials-s3-credentials
        log                                   tls-ca-cert
    
    === 18:49:44 ControllerRevision airflow-scheduled-job/trino-worker-default-78bc6bf4c6  (rev 3, owner StatefulSet/trino-worker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-worker-default-catalog
      ± value change
        - e0695b9e-74d9-4563-a133-24f214378297/1803
        + e0695b9e-74d9-4563-a133-24f214378297/6994
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 481-stackable26.7.0
        + 481-stackable0.0.0-dev
    
    data.spec.template.spec.containers.trino.args.0
      ± value change in multiline text (two inserts, no deletions)
          echo copying /stackable/config to /stackable/rwconfig
          cp -RL /stackable/config/* /stackable/rwconfig
          set +x
          export CATALOG_ICEBERG_S3_AWS_ACCESS_KEY="$(cat /stackable/secrets/minio-s3-credentials/accessKey)"
          export CATALOG_ICEBERG_S3_AWS_SECRET_KEY="$(cat /stackable/secrets/minio-s3-credentials/secretKey)"
          test -f /stackable/rwconfig/exchange-manager.properties && config-utils template /stackable/rwconfig/exchange-manager.properties
          test -f /stackable/rwconfig/spooling-manager.properties && config-utils template /stackable/rwconfig/spooling-manager.properties
        + test -f /stackable/rwconfig/event-listener.properties && config-utils template /stackable/rwconfig/event-listener.properties
          set -x
          
          prepare_signal_handlers()
          {
          
          [25 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          bin/launcher run --etc-dir=/stackable/rwconfig &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
    
    
    data.spec.template.spec.containers.trino.env
      - one list entry removed:                    + three list entries added:
        - name: CONTAINERDEBUG_LOG_DIRECTORY         - name: FILE_LOG_DIRECTORY
        │ value: /stackable/log/containerdebug       │ value: /stackable/log/containerdebug
                                                     - name: FILE_LOG_MAX_FILES
                                                     │ value: 2
                                                     - name: FILE_LOG_ROTATION_PERIOD
                                                     │ value: hourly
    
    data.spec.template.spec.containers.trino.env.INTERNAL_SECRET.valueFrom.secretKeyRef
      - one map entry removed:
        optional: false
    
    data.spec.template.spec.containers.trino.env.SPOOLING_SECRET.valueFrom.secretKeyRef
      - one map entry removed:
        optional: false
    
    data.spec.template.spec.containers.trino.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable26.7.0
        + oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
    
    data.spec.template.spec.containers.trino.volumeMounts
      ⇆ order changed
        server-tls-mount                      config
        server-tls                            rwconfig
        client-tls                            catalog
        internal-tls-mount                    log
        internal-tls                          server-tls-mount
        minio-s3-credentials-s3-credentials   server-tls
        tls-ca-cert                           client-tls
        config                                internal-tls-mount
        rwconfig                              internal-tls
        catalog                               minio-s3-credentials-s3-credentials
        log                                   tls-ca-cert
    data.spec.template.spec.initContainers.prepare.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable26.7.0
        + oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
    
    data.spec.template.spec.initContainers.prepare.volumeMounts
      ⇆ order changed
        server-tls-mount                      rwconfig
        server-tls                            log-config
        client-tls                            log
        internal-tls-mount                    server-tls-mount
        internal-tls                          server-tls
        minio-s3-credentials-s3-credentials   client-tls
        tls-ca-cert                           internal-tls-mount
        rwconfig                              internal-tls
        log-config                            minio-s3-credentials-s3-credentials
        log                                   tls-ca-cert
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes
      ⇆ order changed
        trino-users                           config
        users                                 rwconfig
        server-tls-mount                      catalog
        server-tls                            log-config
        client-tls                            log
        internal-tls-mount                    server-tls-mount
        internal-tls                          server-tls
        minio-s3-credentials-s3-credentials   client-tls
        tls-ca-cert                           internal-tls-mount
        log-config                            internal-tls
        config                                trino-users
        rwconfig                              users
        catalog                               minio-s3-credentials-s3-credentials
        log                                   tls-ca-cert
    
    === 18:49:45 ControllerRevision airflow-scheduled-job/opa-server-default-6f956c4cf7  (rev 2, owner DaemonSet/opa-server-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 1.16.2-stackable26.7.0
        + 1.16.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.opa.args.0
      ± value change in multiline text (one insert, no deletions)
          
          prepare_signal_handlers()
          {
              unset term_child_pid
          
          [24 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          opa run -s -a 0.0.0.0:8081 -c /stackable/config/config.json -l info --shutdown-grace-period 120 --disable-telemetry   &> >(CONSOLE_LEVEL=INFO FILE_LEVEL=INFO DECISION_LEVEL=INFO SERVER_LEVEL=INFO OPA_ROLLING_LOG_FILE_SIZE_BYTES=5000000 OPA_ROLLING_LOG_FILES=2 STACKABLE_LOG_DIR=/stackable/log CONTAINER_NAME=opa process-logs) &
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
      
      
    
    data.spec.template.spec.containers.opa.env
      - one list entry removed:                    + three list entries added:
        - name: CONTAINERDEBUG_LOG_DIRECTORY         - name: FILE_LOG_DIRECTORY
        │ value: /stackable/log/containerdebug       │ value: /stackable/log/containerdebug
                                                     - name: FILE_LOG_MAX_FILES
                                                     │ value: 2
                                                     - name: FILE_LOG_ROTATION_PERIOD
                                                     │ value: hourly
    
    data.spec.template.spec.containers.opa.image
      ± value change
        - oci.stackable.tech/sdp/opa:1.16.2-stackable26.7.0
        + oci.stackable.tech/sdp/opa:1.16.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.bundle-builder.args.0
      ± value change in multiline text (one insert, one deletion)
          
          prepare_signal_handlers()
          {
              unset term_child_pid
          
          [23 lines unchanged)]
          
              set -e
          }
          
          prepare_signal_handlers
        - mkdir -p /stackable/log/bundle-builder
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          stackable-opa-bundle-builder &
          wait_for_termination $!
    
    
    data.spec.template.spec.containers.bundle-builder.env
      ⇆ order changed                                             + two list entries added:
        WATCH_NAMESPACE             CONSOLE_LOG_LEVEL               - name: FILE_LOG_MAX_FILES
        CONSOLE_LOG_LEVEL           FILE_LOG_DIRECTORY              │ value: 5
        FILE_LOG_LEVEL              FILE_LOG_LEVEL                  - name: FILE_LOG_ROTATION_PERIOD
        FILE_LOG_DIRECTORY          KUBERNETES_CLUSTER_DOMAIN       │ value: minutely
        KUBERNETES_NODE_NAME        KUBERNETES_NODE_NAME
        KUBERNETES_CLUSTER_DOMAIN   WATCH_NAMESPACE
    data.spec.template.spec.containers.bundle-builder.image
      ± value change
        - oci.stackable.tech/sdp/opa-operator:26.7.0
        + oci.stackable.tech/sdp/opa-operator:0.0.0-dev
    
    data.spec.template.spec.initContainers.prepare.image
      ± value change
        - oci.stackable.tech/sdp/opa:1.16.2-stackable26.7.0
        + oci.stackable.tech/sdp/opa:1.16.2-stackable0.0.0-dev
    
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    
    === 18:49:51 ControllerRevision airflow-scheduled-job/kafka-broker-default-547969b9d9  (rev 4, owner StatefulSet/kafka-broker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/kafka-broker-default
      ± value change
        - c5664b49-b4c2-4494-901e-2d0d4316bf78/1265
        + c5664b49-b4c2-4494-901e-2d0d4316bf78/7392
    
    
    === 18:49:51 ControllerRevision airflow-scheduled-job/kafka-controller-default-5b9bff66b7  (rev 2, owner StatefulSet/kafka-controller-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/kafka-controller-default
      ± value change
        - 1cf1073a-0a0e-4dc0-9fdf-5d6f743044d3/1304
        + 1cf1073a-0a0e-4dc0-9fdf-5d6f743044d3/7393
    
    
    === 18:49:52 ControllerRevision airflow-scheduled-job/kafka-broker-default-54968c8759  (rev 5, owner StatefulSet/kafka-broker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/kafka
      ± value change
        - fea85e4b-0d10-43a8-b7cf-a0693d9c3362/1426
        + fea85e4b-0d10-43a8-b7cf-a0693d9c3362/7394
    
    
    === 18:49:53 ControllerRevision airflow-scheduled-job/kafka-broker-default-5d45b4b5b9  (rev 6, owner StatefulSet/kafka-broker-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 4.2.1-stackable26.7.0
        + 4.2.1-stackable0.0.0-dev
    
    data.spec.template.spec
      - one map entry removed:
        shareProcessNamespace: true
    
    data.spec.template.spec.containers
      - one list entry removed:
        - name: kcat-prober
        │ image: "oci.stackable.tech/sdp/kafka:4.2.1-stackable26.7.0"
        │ imagePullPolicy: Always
        │ terminationMessagePath: /dev/termination-log
        │ terminationMessagePolicy: File
        │ command:
        │ - sleep
        │ - infinity
        │ resources:
        │ │ limits:
        │ │ │ cpu: 200m
        │ │ │ memory: 128Mi
        │ │ requests:
        │ │ │ cpu: 100m
        │ │ │ memory: 128Mi
        │ volumeMounts:
        │ - name: tls-kcat
        │ │ mountPath: /stackable/tls-kcat
        │ - name: listener-bootstrap
        │ │ mountPath: /stackable/listener-bootstrap
        │ - name: listener-broker
        │ │ mountPath: /stackable/listener-broker
        │ readinessProbe:
        │ │ exec:
        │ │ │ command:
        │ │ │ - /stackable/kcat
        │ │ │ - "-b"
        │ │ │ - "localhost:9093"
        │ │ │ - "-X"
        │ │ │ - security.protocol=SSL
        │ │ │ - "-X"
        │ │ │ - ssl.ca.location=/stackable/tls-kcat/ca.crt
        │ │ │ - "-L"
        │ │ failureThreshold: 3
        │ │ periodSeconds: 2
        │ │ successThreshold: 1
        │ │ timeoutSeconds: 5
        │ env:
        │ - name: POD_NAME
        │ │ valueFrom:
        │ │ │ fieldRef:
        │ │ │ │ apiVersion: v1
        │ │ │ │ fieldPath: metadata.name
    
    data.spec.template.spec.containers.kafka
      + three map entries added:
        livenessProbe:
        │ exec:
        │ │ command:
        │ │ - bash
        │ │ - "-c"
        │ │ - |
        │ │ │ timeout 2 bash -c 'cat < /dev/null > /dev/tcp/localhost/9093' || exit 1
        │ │ │ curl -s --max-time 2 localhost:9606/metrics | grep -qE 'kafka_server_kafkaserver_brokerstate 3(\.0)?$'
        │ failureThreshold: 20
        │ periodSeconds: 30
        │ successThreshold: 1
        │ timeoutSeconds: 10
        readinessProbe:
        │ exec:
        │ │ command:
        │ │ - /stackable/kcat
        │ │ - "-b"
        │ │ - "localhost:9093"
        │ │ - "-X"
        │ │ - security.protocol=SSL
        │ │ - "-X"
        │ │ - ssl.ca.location=/stackable/tls-kcat/ca.crt
        │ │ - "-L"
        │ failureThreshold: 3
        │ periodSeconds: 2
        │ successThreshold: 1
        │ timeoutSeconds: 5
        startupProbe:
        │ exec:
        │ │ command:
        │ │ - bash
        │ │ - "-c"
        │ │ - |
        │ │ │ timeout 2 bash -c 'cat < /dev/null > /dev/tcp/localhost/9093' || exit 1
        │ │ │ curl -s --max-time 2 localhost:9606/metrics | grep -qE 'kafka_server_kafkaserver_brokerstate 3(\.0)?$'
        │ failureThreshold: 60
        │ periodSeconds: 5
        │ successThreshold: 1
        │ timeoutSeconds: 5
    
    data.spec.template.spec.containers.kafka.args.0
      ± value change in multiline text (three inserts, two deletions)
          
          prepare_signal_handlers()
          {
              unset term_child_pid
          
          [24 lines unchanged)]
          
          }
          
          rm -f /stackable/log/_vector/shutdown
          prepare_signal_handlers
        + mkdir --parents "$FILE_LOG_DIRECTORY" || true
          containerdebug --output=/stackable/log/containerdebug-state.json --loop &
          
          
          
          
        - export POD_INDEX=$(echo "$POD_NAME" | grep -oE '[0-9]+$')
        - export REPLICA_ID=$((POD_INDEX+NODE_ID_OFFSET))
        + POD_INDEX=$(echo "$POD_NAME" | grep -oE '[0-9]+$')
        + export REPLICA_ID=$((POD_INDEX + NODE_ID_OFFSET))
          
          if [ -f "/stackable/broker-id-pod-map/$POD_NAME" ]; then
              REPLICA_ID=$(cat "/stackable/broker-id-pod-map/$POD_NAME")
          fi
          
          [four lines unchanged)]
          
          cp /stackable/config/jaas.properties /tmp/jaas.properties
          config-utils template /tmp/jaas.properties
          
          
        - bin/kafka-storage.sh format --cluster-id "$KAFKA_CLUSTER_ID" --config /tmp/broker.properties --ignore-formatted --initial-controllers 2110489703@kafka-controller-default-0.kafka-controller-default-headless.airflow-scheduled-job.svc.cluster.local:9093:0000000000-02110489703
        + bin/kafka-storage.sh format --cluster-id "$KAFKA_CLUSTER_ID" --config /tmp/broker.properties --ignore-formatted --no-initial-controllers
          bin/kafka-server-start.sh /tmp/broker.properties &
          
          
          wait_for_termination $!
          mkdir -p /stackable/log/_vector && touch /stackable/log/_vector/shutdown
      
      
    
    data.spec.template.spec.containers.kafka.env
      ⇆ order changed                             - one list entry removed:                    + three list entries added:
        EXTRA_ARGS          EXTRA_ARGS              - name: CONTAINERDEBUG_LOG_DIRECTORY         - name: FILE_LOG_DIRECTORY
        KAFKA_HEAP_OPTS     KAFKA_CLIENT_PORT       │ value: /stackable/log/containerdebug       │ value: /stackable/log/containerdebug
        KAFKA_LOG4J_OPTS    KAFKA_CLUSTER_ID                                                     - name: FILE_LOG_MAX_FILES
        NODE_ID_OFFSET      KAFKA_HEAP_OPTS                                                      │ value: 2
        KAFKA_CLIENT_PORT   KAFKA_LOG4J_OPTS                                                     - name: FILE_LOG_ROTATION_PERIOD
        KAFKA_CLUSTER_ID    NODE_ID_OFFSET                                                       │ value: hourly
        POD_NAME            POD_NAME
        BOOTSTRAP_SERVER    BOOTSTRAP_SERVER
    data.spec.template.spec.containers.kafka.image
      ± value change
        - oci.stackable.tech/sdp/kafka:4.2.1-stackable26.7.0
        + oci.stackable.tech/sdp/kafka:4.2.1-stackable0.0.0-dev
    
    data.spec.template.spec.containers.kafka.volumeMounts
      + one list entry added:
        - name: tls-kcat
        │ mountPath: /stackable/tls-kcat
    
    data.spec.template.spec.securityContext
      + two map entries added:
        runAsNonRoot: true
        seccompProfile:
        │ type: RuntimeDefault
    
    data.spec.template.spec.volumes.listener-broker.ephemeral.volumeClaimTemplate.metadata.labels.app.kubernetes.io/version
      ± value change
        - 4.2.1-stackable26.7.0
        + 4.2.1-stackable0.0.0-dev
    
    product upgrade changes

    Find the applicable resources (so far just the ones containing "cluster").

    kubectl api-resources --verbs=list --namespaced -o name | grep stackable.tech | \
      xargs -n 1 kubectl get --show-kind --ignore-not-found -n airflow-scheduled-job

    I found the versions based on the stack in main at the time:
    https://github.com/stackabletech/demos/tree/main/stacks/airflow

    kubectl -n airflow-scheduled-job patch opacluster.opa.stackable.tech/opa --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"1.20.2"}]' 
    kubectl -n airflow-scheduled-job patch hivecluster.hive.stackable.tech/hive-iceberg --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"4.2.1"}]' 
    kubectl -n airflow-scheduled-job patch kafkacluster.kafka.stackable.tech/kafka --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"4.3.1"}]' 
    kubectl -n airflow-scheduled-job patch trinocluster.trino.stackable.tech/trino --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"483"}]' 
    kubectl -n airflow-scheduled-job patch airflowcluster.airflow.stackable.tech/airflow --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"3.3.2"}]' 
    === 10:13:02 ControllerRevision airflow-scheduled-job/opa-server-default-c547fc9fb  (rev 3, owner DaemonSet/opa-server-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 1.16.2-stackable0.0.0-dev
        + 1.20.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.opa.image
      ± value change
        - oci.stackable.tech/sdp/opa:1.16.2-stackable0.0.0-dev
        + oci.stackable.tech/sdp/opa:1.20.2-stackable0.0.0-dev
    
    data.spec.template.spec.initContainers.prepare.image
      ± value change
        - oci.stackable.tech/sdp/opa:1.16.2-stackable0.0.0-dev
        + oci.stackable.tech/sdp/opa:1.20.2-stackable0.0.0-dev
    
    
    === 10:15:40 ControllerRevision airflow-scheduled-job/hive-iceberg-metastore-default-78dc44688  (rev 4, owner StatefulSet/hive-iceberg-metastore-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/hive-iceberg-metastore-default
      ± value change
        - 180e76f7-0280-44be-938d-95b6684f43a2/6571
        + 180e76f7-0280-44be-938d-95b6684f43a2/46097
    
    
    === 10:15:40 ControllerRevision airflow-scheduled-job/hive-iceberg-metastore-default-66744b7b48  (rev 5, owner StatefulSet/hive-iceberg-metastore-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 4.2.0-stackable0.0.0-dev
        + 4.2.1-stackable0.0.0-dev
    
    data.spec.template.spec.containers.hive.image
      ± value change
        - oci.stackable.tech/sdp/hive:4.2.0-stackable0.0.0-dev
        + oci.stackable.tech/sdp/hive:4.2.1-stackable0.0.0-dev
    
    
    === 10:17:08 ControllerRevision airflow-scheduled-job/kafka-broker-default-9689874c  (rev 7, owner StatefulSet/kafka-broker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/kafka-broker-default
      ± value change
        - c5664b49-b4c2-4494-901e-2d0d4316bf78/7392
        + c5664b49-b4c2-4494-901e-2d0d4316bf78/46403
    
    
    === 10:17:09 ControllerRevision airflow-scheduled-job/kafka-controller-default-7c6cfff956  (rev 3, owner StatefulSet/kafka-controller-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/kafka-controller-default
      ± value change
        - 1cf1073a-0a0e-4dc0-9fdf-5d6f743044d3/7393
        + 1cf1073a-0a0e-4dc0-9fdf-5d6f743044d3/46405
    
    
    === 10:17:09 ControllerRevision airflow-scheduled-job/kafka-broker-default-8b99dbfbb  (rev 8, owner StatefulSet/kafka-broker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/kafka
      ± value change
        - fea85e4b-0d10-43a8-b7cf-a0693d9c3362/7394
        + fea85e4b-0d10-43a8-b7cf-a0693d9c3362/46409
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 4.2.1-stackable0.0.0-dev
        + 4.3.1-stackable0.0.0-dev
    
    data.spec.template.spec.containers.kafka.image
      ± value change
        - oci.stackable.tech/sdp/kafka:4.2.1-stackable0.0.0-dev
        + oci.stackable.tech/sdp/kafka:4.3.1-stackable0.0.0-dev
    
    data.spec.template.spec.volumes.listener-broker.ephemeral.volumeClaimTemplate.metadata.labels.app.kubernetes.io/version
      ± value change
        - 4.2.1-stackable0.0.0-dev
        + 4.3.1-stackable0.0.0-dev
    
    
    === 10:18:27 ControllerRevision airflow-scheduled-job/trino-coordinator-default-7c968b77f7  (rev 4, owner StatefulSet/trino-coordinator-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-coordinator-default
      ± value change
        - 05edb008-9739-4a15-8df2-a5cb276c53d9/6987
        + 05edb008-9739-4a15-8df2-a5cb276c53d9/46855
    
    
    === 10:18:27 ControllerRevision airflow-scheduled-job/trino-worker-default-688c4dcccc  (rev 4, owner StatefulSet/trino-worker-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-worker-default
      ± value change
        - 4409e7ca-3aa5-4abb-abe7-1e3979d47d7f/6990
        + 4409e7ca-3aa5-4abb-abe7-1e3979d47d7f/46861
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-worker-default-catalog
      ± value change
        - e0695b9e-74d9-4563-a133-24f214378297/6994
        + e0695b9e-74d9-4563-a133-24f214378297/46862
    
    
    === 10:18:28 ControllerRevision airflow-scheduled-job/trino-coordinator-default-5b544576d8  (rev 5, owner StatefulSet/trino-coordinator-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/trino-coordinator-default-catalog
      ± value change
        - 30ee77a9-6807-4339-b40d-e589a05b144f/6988
        + 30ee77a9-6807-4339-b40d-e589a05b144f/46857
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 481-stackable0.0.0-dev
        + 483-stackable0.0.0-dev
    
    data.spec.template.spec.containers.trino.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
        + oci.stackable.tech/sdp/trino:483-stackable0.0.0-dev
    
    data.spec.template.spec.containers.password-file-updater.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
        + oci.stackable.tech/sdp/trino:483-stackable0.0.0-dev
    
    data.spec.template.spec.initContainers.prepare.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
        + oci.stackable.tech/sdp/trino:483-stackable0.0.0-dev
    
    
    === 10:18:29 ControllerRevision airflow-scheduled-job/trino-worker-default-7c4c6d6fdc  (rev 5, owner StatefulSet/trino-worker-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 481-stackable0.0.0-dev
        + 483-stackable0.0.0-dev
    
    data.spec.template.spec.containers.trino.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
        + oci.stackable.tech/sdp/trino:483-stackable0.0.0-dev
    
    data.spec.template.spec.initContainers.prepare.image
      ± value change
        - oci.stackable.tech/sdp/trino:481-stackable0.0.0-dev
        + oci.stackable.tech/sdp/trino:483-stackable0.0.0-dev
    
    
    === 10:19:53 ControllerRevision airflow-scheduled-job/airflow-triggerer-default-54569f77c6  (rev 5, owner StatefulSet/airflow-triggerer-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/47384
    
    
    === 10:19:54 ControllerRevision airflow-scheduled-job/airflow-webserver-default-5d49969f6  (rev 5, owner StatefulSet/airflow-webserver-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/47384
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-webserver-default
      ± value change
        - d08176fc-944b-4f04-b359-f0b88ea02ce7/6649
        + d08176fc-944b-4f04-b359-f0b88ea02ce7/47388
    
    
    === 10:19:54 ControllerRevision airflow-scheduled-job/airflow-dagprocessor-default-6946c495b8  (rev 5, owner StatefulSet/airflow-dagprocessor-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/47384
    
    
    === 10:19:54 ControllerRevision airflow-scheduled-job/airflow-scheduler-default-54fb5695dd  (rev 5, owner StatefulSet/airflow-scheduler-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-executor-pod-template
      ± value change
        - 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/6648
        + 6542d380-3f68-4e5e-ac7a-d0e9af1e21f2/47384
    
    
    === 10:19:54 ControllerRevision airflow-scheduled-job/airflow-scheduler-default-55c4cc9886  (rev 6, owner StatefulSet/airflow-scheduler-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-scheduler-default
      ± value change
        - a23be904-cf45-4d4f-a010-f93951a2d717/6650
        + a23be904-cf45-4d4f-a010-f93951a2d717/47389
    
    
    === 10:19:55 ControllerRevision airflow-scheduled-job/airflow-webserver-default-775d9ff5c4  (rev 6, owner StatefulSet/airflow-webserver-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable0.0.0-dev
        + 3.3.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
        + oci.stackable.tech/sdp/airflow:3.3.2-stackable0.0.0-dev
    
    
    === 10:19:55 ControllerRevision airflow-scheduled-job/airflow-triggerer-default-6f66b748c  (rev 6, owner StatefulSet/airflow-triggerer-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-triggerer-default
      ± value change
        - 9df97e6e-a740-4a9b-b5a4-78ad24ed770e/6660
        + 9df97e6e-a740-4a9b-b5a4-78ad24ed770e/47396
    
    
    === 10:19:56 ControllerRevision airflow-scheduled-job/airflow-dagprocessor-default-5b775484f8  (rev 6, owner StatefulSet/airflow-dagprocessor-default) ===
    
    data.spec.template.metadata.annotations.configmap.restarter.stackable.tech/airflow-dagprocessor-default
      ± value change
        - e2b9d3a7-24f4-4436-b59a-6f294d4d3176/6656
        + e2b9d3a7-24f4-4436-b59a-6f294d4d3176/47392
    
    
    === 10:19:57 ControllerRevision airflow-scheduled-job/airflow-dagprocessor-default-5b667f9f5b  (rev 7, owner StatefulSet/airflow-dagprocessor-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable0.0.0-dev
        + 3.3.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
        + oci.stackable.tech/sdp/airflow:3.3.2-stackable0.0.0-dev
    
    
    === 10:19:58 ControllerRevision airflow-scheduled-job/airflow-scheduler-default-779746f8f8  (rev 7, owner StatefulSet/airflow-scheduler-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable0.0.0-dev
        + 3.3.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
        + oci.stackable.tech/sdp/airflow:3.3.2-stackable0.0.0-dev
    
    
    === 10:19:58 ControllerRevision airflow-scheduled-job/airflow-triggerer-default-d846dc7dc  (rev 7, owner StatefulSet/airflow-triggerer-default) ===
    
    data.spec.template.metadata.labels.app.kubernetes.io/version
      ± value change
        - 3.2.2-stackable0.0.0-dev
        + 3.3.2-stackable0.0.0-dev
    
    data.spec.template.spec.containers.metrics.image
      ± value change
        - oci.stackable.tech/sdp/airflow:3.2.2-stackable0.0.0-dev
        + oci.stackable.tech/sdp/airflow:3.3.2-stackable0.0.0-dev
    
  4. Techassi commented on Nov 10, 2026

    @Techassi
    MemberAuthor

    🟢 signal-processing

    • 🟢 Stable from scratch
    • 🟢 Upgrade to nightly
  5. sbernauer commented on Nov 10, 2026

    @sbernauer
    Member

    🟢 trino-iceberg Just works

  6. Techassi commented on Nov 10, 2026

    @Techassi
    MemberAuthor

    🟢 end-to-end-security

    • 🟢 Stable from scratch
    • 🟢 Upgrade to nightly
  7. adwk67 commented on Nov 10, 2026

    @adwk67
    Member

    🟢 jupyterhub-keycloak

    • 🟢 Stable from scratch
      • selected namespace is not picked up, is using default instead: has to be installed in default as the change allowing other namespaces was not made until later
    • 🟢 Upgrade to nightly
  8. sbernauer commented on Nov 11, 2026

    @sbernauer
    Member

    🟢 trino-taxi-data just works

  9. adwk67 commented on Nov 11, 2026

    @adwk67
    Member

    🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

    • 🟢 Stable from scratch
    • 🟢 Upgrade to nightly

    Product bumps (in addition to op bumps)

    Stacklet Before After
    ZooKeeper 3.9.5 3.9.6
    Spark 4.1.2 4.2.0

    (spark: the resource was updated but the pods were not restarted for some reason)
    Seems to be fixed with: #480

  10. razvan commented on Nov 11, 2026

    @razvan
    Member

    Demos tested

    • 🟢 nifi-kafka-druid-earthquake-data
    • 🟢 nifi-kafka-druid-water-level-data

    Results

    • 🟢 Stable from scratch
    • 🟢 Update to nightly

    Product bumps (in addition to op bumps)

    Stacklet Before After
    ZooKeeper 3.9.5 3.9.6
    Kafka 4.2.1 4.3.1
    NiFi 2.9.0 2.12.0

    Pull request

    Describes how to flush Druid segments immediately without having to wait an hour: #479

  11. moved this from Development: In Progress to Done in Stackable Engineeringon Nov 11, 2026
  12. moved this from Done to Development: In Progress in Stackable Engineeringon Nov 11, 2026
  13. adwk67 commented on Nov 11, 2026

    @adwk67
    Member

    🟢 spark-k8s-anomaly-detection-taxi-data

    • 🟢 Stable from scratch
    • 🟢 Upgrade to nightly

    Product bumps (in addition to op bumps)

    Stacklet Before After
    hive 4.0.0 4.2.1
    opa 1.16.2 1.20.2
    trino 481 483
  14. adwk67 commented on Nov 11, 2026

    @adwk67
    Member

    🟢 hbase-hdfs-load-cycling-data

    • 🟢 Stable from scratch
    • 🟢 Upgrade to nightly
    • 🟢 HDFS rolling upgrade
      • no product version change between 26.7 and main, so...
      • ..tested on an integration test from 26.7 using 3.4.3, upgraded operators, followed docs

    Product bumps (in addition to op bumps)

    Stacklet Before After
    ZooKeeper 3.9.5 3.9.6
    HBase 2.6.6 2.6.7
  15. Techassi commented on Nov 12, 2026

    @Techassi
    MemberAuthor

    🟢 tutorial-openldap (stack)

    • 🟢 Stable from scratch
    • 🟢 Upgrade to nightly
  16. adwk67 commented on Nov 12, 2026

    @adwk67
    Member

    🟢 argo-cd-git-ops

    • 🟢 Stable from scratch

    Testing this demo against a release is tricky as not only is release-26.7 using minio, but customGitBranch defaults to main (demos-v2.yaml:41) as do sealed-secrets.yaml:31 and the DAG git-sync (airflow.yaml:30). Using customGitBranch=release-26.7 would not have been sufficient (unless garage was patched into a temporary branch based off of the release) as the names of the init containers changed and they were patched to bump up resources. I started the demo with --parameters customGitBranch=fix/argo-cd-pre-26.11 --stack-parameters stackableReleaseVersion=26.7.0 but also had to get Claude to pause ArgoCd and patch various things to get manifest running properly with 26.7. Eventually it was all up and running.

  17. Techassi commented on Nov 13, 2026

    @Techassi
    MemberAuthor

    Done, closing.

  18. moved this from Development: In Progress to Development: Done in Stackable Engineeringon Nov 13, 2026
  19. moved this from Development: Done to Done in Stackable Engineeringon Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions