John Ramirez
05/08/2020, 4:01 PMDaskKubernetesEnvironment
to run my workflows. There will be times where I will need to submit a high volume of flows to the cluster. My expectation is that the auto-scaler will spin up the additional EC2 instances required to handle to temporary volume increase but I am concerned about the latency (that is the time it takes to make the additional resources available). Is there a setting on the retry the entire flow if it fails? That way it reduces the gap.Jenny
05/08/2020, 4:25 PMJohn Ramirez
05/08/2020, 5:37 PMJenny
05/08/2020, 6:18 PMfrom dask.distributed import get_client
@task
def wait_for_resources():
client = get_client()
# Wait until you have 10 workers
client.wait_for_workers(n_workers=10)
• Or if you're concerned only about stopping the flow run from failing, you could use a root task that waits for resources to become available before finishing - if you set your other tasks as dependent e.g. by using a trigger, they won't run until it's finished
Also worth noting that flow-run-concurrency limiting is a roadmap item so you should be able to configure this natively at some point in the future.