Joseph Mathes

06/14/2022, 2:25 PM
When running a flow from prefect cloud using a Coiled Dask executor, I'm noticing long waits (on the order of minutes) before the first task starts running. Is that expected? Are there any best practices (or even not-so-best practices) that can be used to shorten that loop? I'm hoping to make it part of a dev iteration cycle, which would make those two minutes pretty common, and thus expensive I'm also asking the Coiled people if there are any options on their side

Kevin Kho

06/14/2022, 2:31 PM
Yes expected just because that’s the Dask cluster spinup side (if you see the same delay for
). They would be the right ones to ask.
On the Prefect side, I think you can decrease the ECS spin up time by backing it with an EC2 instance that can cache the image?