John Ramirez05/08/2020, 4:01 PM
to run my workflows. There will be times where I will need to submit a high volume of flows to the cluster. My expectation is that the auto-scaler will spin up the additional EC2 instances required to handle to temporary volume increase but I am concerned about the latency (that is the time it takes to make the additional resources available). Is there a setting on the retry the entire flow if it fails? That way it reduces the gap.
Jenny05/08/2020, 4:25 PM
John Ramirez05/08/2020, 5:37 PM
Jenny05/08/2020, 6:18 PM
• Or if you're concerned only about stopping the flow run from failing, you could use a root task that waits for resources to become available before finishing - if you set your other tasks as dependent e.g. by using a trigger, they won't run until it's finished Also worth noting that flow-run-concurrency limiting is a roadmap item so you should be able to configure this natively at some point in the future.
from dask.distributed import get_client @task def wait_for_resources(): client = get_client() # Wait until you have 10 workers client.wait_for_workers(n_workers=10)