is it possible to use a LocalDaskExecutor with an ECS run config? i’m a bit confused on how i can leverage Dask and if it requires setting up any additional AWS IAM policies or resources to run in conjunction with my ECS run config. basically right now i have a complex flow with ~200 tasks and i’d like to increase the number of tasks that can be executed as one time.

Yes you can use LocalDask but from experience, just specify your num_workers and threads otherwise it will default to 2. You would give your ECS task a role that would give it the resources to spin up a cluster. The DaskExecutor either creates a new cluster or connects to an existing one so it will be separate compute from the ECS container

just specify your num_workers and threads otherwise it will default to 2

to clarify, are you suggesting i try

LocalDaskExecutor

before

DaskExecutor

and just configure my workers/threads? do i need to update any IAM policy to run

LocalDaskExecutor

?

You would give your ECS task a role that would give it the resources to spin up a cluster

would my ECS task spin up the cluster, or the ECS service running the ECS agent?

You don’t need IAM because LocalDaskExecutor is just like a multiprocessing pool

i added a LocalDaskExecutor to the flow but the metadata is unchanged when i register it. is that expected? where can i confirm in the prefect UI that it’s using that executor?

that is actually expected because executor is not part of the metadata that we store in the backend because it contain stuff like the Dask cluster address so we just read it from the Flow storage. You will need to force the registration for it to take effect, and you need it in the same file as the Flow

so i set the executor by doing this in my flow code:

flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)

does it need anything else? i see when i try to kick it off now, in the ECS task logs:

/bin/sh: 1: prefect: not found

I think that is a separate issue from the executor right? Did it work without the executor and what image are you using?

yeah it worked just before this. i’m using a task definition arn which includes a

flow

image with prefect installed

I can’t think of anything immediately beyond asking if you could try without LocalDask to confirm

maybe i’ll try restarting the flow again? i see this in the prefect logs

Rescheduled by a Lazarus process. This is attempt 1.

Do you have more logs and did you use the base Prefect image? Yeah you can try that

i use my own custom image that’s just the base prefect image with some additional requirements installed

Yeah that makes sense. That should work

okay i removed the executor and got the same error. so i guess it’s something else (even though i didn’t change anything 😭). thanks for your help

Do you have CloudWatch logs enabled?

we use datadog for our logging. i assume they’re the same logs though

Ah that makes sense.