https://prefect.io logo
Title
m

Mike Vanbuskirk

07/22/2022, 10:14 PM
do you absolutely have to explicitly specify an ECS cluster name to an
ECSRun
config or can it not infer the correct cluster from providing agent labels?
1
m

Mathijs Miermans

07/22/2022, 10:25 PM
I believe it will use a cluster name of
"default"
if it's not provided.
`cluster (str, optional)`: The AWS cluster to use, defaults to
"default"
if not provided.
https://docs.prefect.io/api/latest/agent/ecs.html#ecsagent
We provide the cluster name to the agent using the
--cluster
command line argument, and don't set it in
ECSRun
, and that works for us.
m

Mike Vanbuskirk

07/22/2022, 10:35 PM
🤔
I think the compositional structure of the Terraform module that Prefect created kind of makes labels redundant if you can’t use them to infer cluster
is the
--cluster
option part of the
prefect cli
?
m

Mathijs Miermans

07/22/2022, 10:43 PM
Yeah, it an argument to `agent ecs start`: https://docs.prefect.io/api/latest/cli/agent.html#ecs-start
m

Mike Vanbuskirk

07/22/2022, 10:44 PM
erm… huh, k… our agents are already running on the cluster… not sure where else they would run… I’m talking about flow jobs specifically
seems like a data eng should not have to care about infra when writing flow jobs
m

Mathijs Miermans

07/22/2022, 10:49 PM
It would be nice if the default behavior for the ECS agent was to start flows on the same ECS cluster as the agent was running on. I remember running into this early on.
m

Mike Vanbuskirk

07/22/2022, 10:49 PM
agreed
a

Anna Geller

07/22/2022, 11:26 PM
if you need more ECS examples, check out this repo e.g. here https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows/ecs_flows_from_5411_pr you don't have to specify cluster name, e.g. here it will be inferred from the agent:
RUN_CONFIG = ECSRun(
    labels=["ecs-agent-local"],
    task_definition_path="<s3://prefectdata/flow_task_definition.yaml>",
)
m

Mike Vanbuskirk

07/22/2022, 11:32 PM
awesome, thank you! re: inferring cluster, it may be an artifact of the TF config provided
👍 1