If you use something off-cluster to store logs like Papertrail or similar, this won’t be an issue.. but

kubectl logs

won’t help you

Having a long running cluster may certainly help. You could go to the Dask dashboard and look for logs there. Some people to ship this out to third party services to retain worker logs somewhere

this kinda stuff is easier to setup once. I.e you can configure the k8s cluster with a fluentd sidecar and grab stdout from all pods. IDK if GCP has something better

@Evan Curtin Interesting. I’ll explore the fluentd sidecar option as well. Thanks!

Basically you write a

cron

expression to describe when you want it to be up

…and here’s a blog about doing it: https://medium.com/google-cloud/scale-your-kubernetes-cluster-to-almost-zero-with-gke-autoscaler-9c78051cbf40

@Wilson Bilkovich The blogpost seems pretty straight-forward. Just wondering if you think I’d need special handling in order to get this to play nice with KubeCluster, which appears to use a default pod template for dask scheduler and workers: https://kubernetes.dask.org/en/latest/_modules/dask_kubernetes/core.html#KubeCluster Wondering if the use of labels for the node-pool (and a few other yaml customizations for pods) would require providing custom pod templates (for dask scheduler and workers) to KubeCluster in order to ensure the job is run on the burst pool (as opposed to the default node pool, which would presumably run the Prefect agent)

?

You will want to specify custom pod templates for Dask I’d say though, so you can do the “anti-affinity” stuff the bblog post mentions

Something like this but with different numbers and names

pod_spec = make_pod_spec(
    image="daskdev/dask:latest",
    memory_limit="4G",
    memory_request="4G",
    cpu_limit=1,
    cpu_request=1,
    env={"EXTRA_PIP_PACKAGES": "prefect dask-kubernetes"},
)

executor = DaskExecutor(
    cluster_class=KubeCluster,
    cluster_kwargs=dict(
        namespace="prefect",
        pod_template=pod_spec,
        n_workers=1,
    ),
)

Some of the things in the blog post are optional; for example the anti-affinity that keeps multiple workers off the same node, you could probably live without that

I think this conversation has talked me into doing it this way as well.. I’ll try configuring a second node group on my k8s cluster, for Dask