https://prefect.io
Join Slack
do you absolutely have to explicitly specify an ECS cluster name to an `ECSRun` config or can it not...
# ask-community
m
do you absolutely have to explicitly specify an ECS cluster name to an 
ECSRun
config or can it not infer the correct cluster from providing agent labels?
✅ 1
m
I believe it will use a cluster name of 
"default"
if it's not provided.
`cluster (str, optional)`: The AWS cluster to use, defaults to 
"default"
if not provided.
https://docs.prefect.io/api/latest/agent/ecs.html#ecsagent
We provide the cluster name to the agent using the 
--cluster
command line argument, and don't set it in 
ECSRun
, and that works for us.
m
🤔
I think the compositional structure of the Terraform module that Prefect created kind of makes labels redundant if you can’t use them to infer cluster
is the 
--cluster
option part of the 
prefect cli
?
m
Yeah, it an argument to `agent ecs start`: https://docs.prefect.io/api/latest/cli/agent.html#ecs-start
m
erm… huh, k… our agents are already running on the cluster… not sure where else they would run… I’m talking about flow jobs specifically
seems like a data eng should not have to care about infra when writing flow jobs
m
It would be nice if the default behavior for the ECS agent was to start flows on the same ECS cluster as the agent was running on. I remember running into this early on.
m
agreed
a
if you need more ECS examples, check out this repo e.g. here https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows/ecs_flows_from_5411_pr you don't have to specify cluster name, e.g. here it will be inferred from the agent:
Copy code
RUN_CONFIG = ECSRun(
    labels=["ecs-agent-local"],
    task_definition_path="<s3://prefectdata/flow_task_definition.yaml>",
)
m
awesome, thank you! re: inferring cluster, it may be an artifact of the TF config provided
👍 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by