SparkApplication timeToLiveSeconds

[nomnom88]

In the stackable documentation, specifically on this page: https://docs.stackable.tech/home/stable/airflow/usage-guide/applying-custom-resources

There is a SparkKubernetesSensor which contains the string: "Make sure timeToLiveSeconds is set on your SparkApplication spec.\n"

Is there such a property? I don't see it in the CRD. Or is there an other way to set a TTL on my SparkApplication resources so the driver pod isn't immediately destroyed on completion?

[adwk67]

To answer your first question: the SparkKubernetesSensor.py file is provided by the Airflow Kubernetes Provider to work with the Google Spark operator (which it mentions in the comments e.g. here). It works with the Stackable one as well, and this example shows how to do so. The code is shown as-is, which means that the comments refer to the Google operator.

[adwk67]

For the driver TTL: you can use PodOverrides to set this. I'll try and find and example of doing that.

[adwk67]

The standard TTL on the spark Job create by the Stackable Spark operator is 600s, and that on the driver is a lot longer: I've just run a integration test and the driver is not cleaned up automatically, so this (immediate destruction on completion) looks like it is a function of the airflow code. If you are interested, I did a (hopefully objective!) comparison of the two operators a while ago, here, although it will be out-of-date now as the google operator has been taken over by Kubeflow.

[nomnom88]

Thanks @adwk67 , I just found the GCP version in the Airflow CNCF provider github code. This is not mentioned in the stackable docs and I wasn't even aware that the Airflow CNCF Kubernetes Provider also includes a bit of Spark functionality.

Thanks for the clarification.