I am facing an issue autoscaling from 0 nodes using the aws

Question

I am facing an issue autoscaling from 0 nodes using the aws auto scaler on EKS note the issue only arises when autoscaling from 0 and not 1 node I see someone recommended using something like this for waiting on dask worker pods ``` task def wait for resources client = get client Wait until we have 10 workers client wait for workers n workers=10 ``` but this doesn t seem to work for waiting on the first node to be present Has anyone had the chance to try out auto scaling from 0

Jenny · Answer

Thanks for the question < Marwan Sarieddine> I ve not tried this but will see if others in the team have any ideas

Marwan Sarieddine · Answer

Thanks < Jenny> still trying to figure out things with prefect any tips would be appreciated

Jenny · Answer

Hi < Marwan Sarieddine> one immediate thought is that it wouldn t be possible for a prefect task to wait for nodes to go from 0 gt 1 since there needs to be at least 1 node in order for the flow to even start

Marwan Sarieddine · Answer

Hi < Jenny> the scheduler and agent are running on a different node group than the dask workers i e in the dask worker spec I specify a node affinity to another node group

Jenny · Answer

If that doesn t help can you give us a bit more information about the issue it creates for you If the autoscaler is on and configured then once the k8s agent creates the prefect job it should start scaling up

josh · Answer

< Marwan Sarieddine> Which environment are you using for your flow

Marwan Sarieddine · Answer

HI < josh> I am using a `DaskKubernetesEnvironment` I guess to better explain my thinking I was hoping that using prefect I can have a separation of concerns i e the agent scheduler what is creating the work to be running on one node group and the workers what is actually doing the work to run on a separate node group this would allow for one to use the same cluster and multiple worker node groups with different instance types to run different flows with potentially different resource requirements

josh · Answer

Ah I see what you re saying I m not entirely sure how dask kubernetes handles having the scheduler and workers on different node groups Looping in < Jim Crist Harif> in case he has an idea slightly smiling face

Jim Crist-Harif · Answer

Sorry I m having a hard time parsing this issue dask kubernetes supports clusters running in different node groups from the client or schedulers running in different node groups than the workers than the client None of this should be causing issues Can you restate what you re expecting hoping to happen here with your setup and how that differs from what you re seeing

Marwan Sarieddine · Answer

Hi < Jim Crist Harif> sorry if I wasn t so clear communicating the issue what you are saying is true because when I am autoscaling from a nodegroup with a minSize of 1 things work fine The setup is to have the dask workers select a node group that autoscales from a minSize of 0 instead of 1 What ends up happening is since the node is not ready the scheduling of the worker pod is being killed prematurely please see the event logs below for some more insight into the problem note the event logs start right after the flow is run

Marwan Sarieddine · Answer

specifically please see the last two event logs

Marwan Sarieddine · Answer

```4m20s Normal Scheduled pod prefect job 815860a0 gzhl6 Successfully assigned default prefect job 815860a0 gzhl6 to ip 192 168 35 156 us west 2 compute internal 4m20s Normal SuccessfulCreate job prefect job 815860a0 Created pod prefect job 815860a0 gzhl6 4m20s Normal Pulling pod prefect job 815860a0 gzhl6 Pulling image 4m17s Normal Pulled pod prefect job 815860a0 gzhl6 Successfully pulled image 4m17s Normal Created pod prefect job 815860a0 gzhl6 Created container flow 4m16s Normal SuccessfulCreate job prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de Created pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de vs587 4m16s Normal Started pod prefect job 815860a0 gzhl6 Started container flow 4m16s Normal Scheduled pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de vs587 Successfully assigned default prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de vs587 to ip 192 168 17 18 us west 2 compute internal 4m15s Normal Pulled pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de vs587 Container image already present on machine 4m15s Normal Created pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de vs587 Created container flow 4m14s Normal Started pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de vs587 Started container flow 4m4s Normal Scheduled pod dask root 7c45f7c9 art6xn Successfully assigned default dask root 7c45f7c9 art6xn to ip 192 168 35 156 us west 2 compute internal 4m3s Normal Created pod dask root 7c45f7c9 art6xn Created container dask worker 4m3s Normal Pulled pod dask root 7c45f7c9 art6xn Container image already present on machine 4m3s Normal Started pod dask root 7c45f7c9 art6xn Started container dask worker 4m Normal Killing pod dask root 7c45f7c9 art6xn Stopping container dask worker 3m53s Normal Scheduled pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 Successfully assigned default prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 to ip 192 168 35 156 us west 2 compute internal 3m53s Normal SuccessfulCreate job prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de Created pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 3m52s Normal Pulled pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 Container image already present on machine 3m52s Normal Created pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 Created container flow 3m51s Normal Started pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 Started container flow 3m3s Warning FailedScheduling pod dask root 0f923f19 62vfb8 0 2 nodes are available 2 node s didn t match node selector 3m33s Normal TriggeredScaleUp pod dask root 0f923f19 62vfb8 pod triggered scale up eksctl prefect eks test nodegroup eks cpu 2 NodeGroup 1PDZRBRX9TE4I 0 gt 1 max 10 3m33s Normal Killing pod prefect dask job fb5921cb d719 4962 b1ea f0fe5539c8de f4wc9 Stopping container flow 2m52s Normal NodeHasNoDiskPressure node ip 192 168 81 183 us west 2 compute internal Node ip 192 168 81 183 us west 2 compute internal status is now NodeHasNoDiskPressure 2m52s Normal NodeHasSufficientMemory node ip 192 168 81 183 us west 2 compute internal Node ip 192 168 81 183 us west 2 compute internal status is now NodeHasSufficientMemory 2m52s Normal NodeAllocatableEnforced node ip 192 168 81 183 us west 2 compute internal Updated Node Allocatable limit across pods 2m52s Normal NodeHasSufficientPID node ip 192 168 81 183 us west 2 compute internal Node ip 192 168 81 183 us west 2 compute internal status is now NodeHasSufficientPID 2m49s Warning FailedScheduling pod dask root 0f923f19 62vfb8 0 3 nodes are available 1 node s had taints that the pod didn t tolerate 2 node s didn t match node selector 2m53s Normal Starting node ip 192 168 81 183 us west 2 compute internal Starting kubelet 2m51s Normal RegisteredNode node ip 192 168 81 183 us west 2 compute internal Node ip 192 168 81 183 us west 2 compute internal event Registered Node ip 192 168 81 183 us west 2 compute internal in Controller 2m48s Normal Starting node ip 192 168 81 183 us west 2 compute internal Starting kube proxy 2m33s Warning FailedScheduling pod dask root 0f923f19 62vfb8 skip schedule deleting pod default dask root 0f923f19 62vfb8 2m32s Normal NodeReady node ip 192 168 81 183 us west 2 compute internal Node ip 192 168 81 183 us west 2 compute internal status```

Marwan Sarieddine · Answer

if I rerun the flow and that single spawned node is now ready the flow runs fine

Marwan Sarieddine · Answer

I think this warning ```2m49s Warning FailedScheduling pod dask root 0f923f19 62vfb8 0 3 nodes are available 1 node s had taints that the pod didn t tolerate 2 node s didn t match node selector ``` the taint effect is probably NoSchedule because the node is still not ready hence killing the dask worker pod prematurely

Jim Crist-Harif · Answer

Can you share your dask kubernetes configuration I m struggling to find what s triggering that pod deletion afaict we don t have timeouts anywhere for pod startups

Marwan Sarieddine · Answer

sure should I share the worker spec

Jim Crist-Harif · Answer

Yeah and if you re running the scheduler remote or local

Marwan Sarieddine · Answer

remote scheduler and please see the worker spec below ```kind Pod metadata labels app prefect dask worker spec replicas 2 restartPolicy Never imagePullSecrets name gitlab secret note I tried using both affinity and a selector nodeSelector role supplement affinity nodeAffinity requiredDuringSchedulingIgnoredDuringExecution nodeSelectorTerms matchExpressions key role operator In values supplement containers image <http registry gitlab com xxxx|registry gitlab com xxxx> imagePullPolicy IfNotPresent args dask worker nthreads 1 no bokeh memory limit 4GB name dask worker env name AWS BUCKET value xxxx name AWS ACCESS KEY ID valueFrom secretKeyRef name aws secret key AWS ACCESS KEY ID name AWS SECRET ACCESS KEY valueFrom secretKeyRef name aws secret key AWS SECRET ACCESS KEY resources limits cpu 2000m memory 4G requests cpu 1000m memory 2G```

Jim Crist-Harif · Answer

What happens with your `wait for resources ` task Does it hang forever Or does it error Or return early but you don t have workers

Marwan Sarieddine · Answer

it just hangs forever

Marwan Sarieddine · Answer

actually none of the flow tasks look like they are triggered including wait for resources

Jim Crist-Harif · Answer

You re saying that `wait for resources` hasn t been called at all Then you re likely blocking on creating the initial scheduler pod not any of the workers

Marwan Sarieddine · Answer

```22 May 2020 01 54 21 prefect CloudFlowRunner INFO Beginning Flow run for Data Processing 22 May 2020 01 54 21 prefect CloudFlowRunner INFO Starting flow run 22 May 2020 01 54 21 prefect CloudFlowRunner DEBUG Flow Data Processing Handling state change from Scheduled to Running```

Marwan Sarieddine · Answer

these are the logs yes it hasn t been called at all

Marwan Sarieddine · Answer

I see how s that possible if the logs show a state change so I guess that s being triggered by the agent correct

Jim Crist-Harif · Answer

The flow runner has started but since the executor isn t running no task runs have been started

Jim Crist-Harif · Answer

Can you share your scheduler configuration as well

Jim Crist-Harif · Answer

Mainly interested in kwargs you re passing to the KubeCluster

Marwan Sarieddine · Answer

I am not specifying job yaml just using the defualt

Jim Crist-Harif · Answer

Do you set scheduler service wait timeout

Marwan Sarieddine · Answer

i e here is my `DaskKubernetesEnvironment` call ```Flow Data Processing environment=DaskKubernetesEnvironment worker spec file= worker spec yaml min workers=1 max workers=10 storage=Docker registry url=os environ GITLAB REGISTRY image name= dask k8s flow image tag= 0 1 0 python dependencies= boto3==1 13 14 numpy==1 18 4 result=s3 result ```

Jim Crist-Harif · Answer

ok thanks

Marwan Sarieddine · Answer

gt Do you set scheduler service wait timeout As you can see No I don t explicitly set it

Jim Crist-Harif · Answer

You said you re using a remote scheduler did you set that as part of your dask config then

Jim Crist-Harif · Answer

Jim Crist-Harif · Answer

By default deploy mode is local

Marwan Sarieddine · Answer

I am sorry if by local you mean scheduler also runs on the cluster then that is what I intend

Jim Crist-Harif · Answer

There are two deploy modes local means the scheduler runs in the same process as your prefect flow runner which is on k8s remote means the scheduler gets its own pod

Marwan Sarieddine · Answer

I see

Jim Crist-Harif · Answer

Are those logs from your flow run the only output you see from the pod running your flow

Marwan Sarieddine · Answer

thats the output of calling `kubectl get events` I also shared the prefect logs which basically just state that flow runner has started

Marwan Sarieddine · Answer

not sure what other output to inspect in this case

Jim Crist-Harif · Answer

If the pod is still running it d be useful to see `kubectl logs that pod name` Prefect only stores its own logs to cloud there may be other things written to stdout that are missed

Marwan Sarieddine · Answer

I see sure will try to look into it only annoying thing is the pods are very ephemeral sounds like the main culprit is the scheduler timing out prematurely

Marwan Sarieddine · Answer

probably should remove the resource manager to get easier access to the pod logs

Jim Crist-Harif · Answer

If you re running with `deploy mode= local ` which it sounds like you are here s what I think has happened The pod running your flow runner starts The flow runner starts The flow runner creates a dask kubernetes cluster Since you re running with deploy mode local this creates a scheduler process in the same pod The flow runner submits work to the dask scheduler which happily accepts it but has no workers to run things on The scheduler requests worker pods from kubernetes Things hang

Marwan Sarieddine · Answer

exactly

Jim Crist-Harif · Answer

It s a bit hard to track what of the event log above is due to this flow vs other flows but if that last pod that was deleted was for the flow runner it s not clear why it would have been deleted We don t impose time limits on spinning up workers

Jim Crist-Harif · Answer

How long have you let this run before killing it Has the node spun up and sat for a few minutes idle with dask not submitting new workers to it

Marwan Sarieddine · Answer

I let it stay running for at least 10 minutes

Jim Crist-Harif · Answer

That should be easily enough time Something odd has happened

Marwan Sarieddine · Answer

question The scheduler requests worker pods from kubernetes if the kubernetes cluster takes a while to provide the pods does the scheduler time out and get killed

Jim Crist-Harif · Answer

The scheduler is running in the local pod and should hang indefinitely when it has work scheduled but no where to run it It will try to get new workers spun up but shouldn t time out at any point

Marwan Sarieddine · Answer

Oh wow I just removed the resource manager and re ran to get the pod logs i e prevent pods getting killed and it looks like ran just fine

Jim Crist-Harif · Answer

A new node spun up workers were allocated and the flow ran

Marwan Sarieddine · Answer

yep

Marwan Sarieddine · Answer

looks like the resource manager is killing the pod prematurely

Marwan Sarieddine · Answer

removing the resource manager is the only change I made here so I am out of ideas but i think it is causing this issue

Jim Crist-Harif · Answer

Oh wow yep that s it That code deletes pending pods willy nilly That s a bug

josh · Answer

I would buy that the resource manager is causing an issue It has a very greedy removal of resources

Jim Crist-Harif · Answer

Nice find

Jim Crist-Harif · Answer

I ll submit a fix asap

josh · Answer

Thanks < Jim Crist Harif>

Marwan Sarieddine · Answer

< Jim Crist Harif> thank you so much for helping me debug this