https://prefect.io logo
Title
l

Lana Dann

03/23/2022, 3:43 PM
is it possible to use a LocalDaskExecutor with an ECS run config? i’m a bit confused on how i can leverage Dask and if it requires setting up any additional AWS IAM policies or resources to run in conjunction with my ECS run config. basically right now i have a complex flow with ~200 tasks and i’d like to increase the number of tasks that can be executed as one time.
k

Kevin Kho

03/23/2022, 3:48 PM
Yes you can use LocalDask but from experience, just specify your num_workers and threads otherwise it will default to 2. You would give your ECS task a role that would give it the resources to spin up a cluster. The DaskExecutor either creates a new cluster or connects to an existing one so it will be separate compute from the ECS container
l

Lana Dann

03/23/2022, 3:53 PM
just specify your num_workers and threads otherwise it will default to 2
to clarify, are you suggesting i try
LocalDaskExecutor
before
DaskExecutor
and just configure my workers/threads? do i need to update any IAM policy to run
LocalDaskExecutor
?
You would give your ECS task a role that would give it the resources to spin up a cluster
would my ECS task spin up the cluster, or the ECS service running the ECS agent?
k

Kevin Kho

03/23/2022, 3:55 PM
You don’t need IAM because LocalDaskExecutor is just like a multiprocessing pool
:gratitude-thank-you: 1
ECS task running the flow will spin up the cluster
👍 1
l

Lana Dann

03/23/2022, 4:25 PM
i added a LocalDaskExecutor to the flow but the metadata is unchanged when i register it. is that expected? where can i confirm in the prefect UI that it’s using that executor?
k

Kevin Kho

03/23/2022, 4:28 PM
that is actually expected because executor is not part of the metadata that we store in the backend because it contain stuff like the Dask cluster address so we just read it from the Flow storage. You will need to force the registration for it to take effect, and you need it in the same file as the Flow
l

Lana Dann

03/23/2022, 4:54 PM
so i set the executor by doing this in my flow code:
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=8)
does it need anything else? i see when i try to kick it off now, in the ECS task logs:
/bin/sh: 1: prefect: not found
k

Kevin Kho

03/23/2022, 4:58 PM
I think that is a separate issue from the executor right? Did it work without the executor and what image are you using?
l

Lana Dann

03/23/2022, 5:00 PM
yeah it worked just before this. i’m using a task definition arn which includes a
flow
image with prefect installed
k

Kevin Kho

03/23/2022, 5:03 PM
I can’t think of anything immediately beyond asking if you could try without LocalDask to confirm
l

Lana Dann

03/23/2022, 5:05 PM
maybe i’ll try restarting the flow again? i see this in the prefect logs
Rescheduled by a Lazarus process. This is attempt 1.
k

Kevin Kho

03/23/2022, 5:05 PM
Do you have more logs and did you use the base Prefect image? Yeah you can try that
l

Lana Dann

03/23/2022, 5:13 PM
i use my own custom image that’s just the base prefect image with some additional requirements installed
just retried and that’s the only log
/bin/sh: 1: prefect: not found
i’ll try removing and retrying now
k

Kevin Kho

03/23/2022, 5:14 PM
Yeah that makes sense. That should work
l

Lana Dann

03/23/2022, 5:36 PM
okay i removed the executor and got the same error. so i guess it’s something else (even though i didn’t change anything 😭). thanks for your help
k

Kevin Kho

03/23/2022, 5:38 PM
Do you have CloudWatch logs enabled?
l

Lana Dann

03/23/2022, 5:39 PM
we use datadog for our logging. i assume they’re the same logs though
k

Kevin Kho

03/23/2022, 5:40 PM
Ah that makes sense.