Can you point https://docs.prefect.io/orchestration/flow_config/executors.html#connecting-to-an-existing-cluster to a ECS cluster? I.E I have an ECS Agent deployed as a task in a AWS ECS Cluster, can I point the DaskExecutor to the same cluster to contain everything?

yep!

Incoming dumb question - What would the address be? Or do I have to specifically deploy dask first into the ECS Cluster?

I recommend using a temporary dask cluster rather than connecting to an existing dask cluster, but in general yes you'd need to know the address beforehand (there are some hacky ways around this, but I won't recommend them unless you definitely need them).

What we're hoping for is keeping it within a contained space so we can inherit AWS Tagging for costing purposes

For a temporary cluster, you could configure your

DaskExecutor

to use

dask_cloudprovider.aws.FargateCluster

to spin up a temporary dask cluster per flow run.

Oh I'd just pass

tags: <dict>

to the Executor?

Any argument to

FargateCluster

is available here through

cluster_kwargs

, so if

dask_cloudprovider

supports what you need then yes.

Huh. Neat,

No, you'd need to install it extra in your image.

Hmmm okay, would including it in the base image be a bad thing?

Sorry, what base image?

Maybe I'm missing something. What image would dask use if I didn't specify one?

Gotcha. If your flow(s) make use of it, then you'll need to install it in the image specified on your flow's run_config.

I guess my initial thought is why isn't that just in the prefect image, not to sound too blunt haha https://docs.prefect.io/orchestration/flow_config/executors.html#executors could do with being more explicit in saying 'you can't run this example unless you install X,Y,Z with :W'

ah, gotcha. Dask has many many different deployment backends - we can't include them all, so we opt not to include any of them.

Fair enough haha