Mike Vanbuskirk

    Mike Vanbuskirk

    2 months ago
    do you absolutely have to explicitly specify an ECS cluster name to an
    ECSRun
    config or can it not infer the correct cluster from providing agent labels?
    Mathijs Miermans

    Mathijs Miermans

    2 months ago
    I believe it will use a cluster name of
    "default"
    if it's not provided.
    cluster (str, optional): The AWS cluster to use, defaults to
    "default"
    if not provided.
    https://docs.prefect.io/api/latest/agent/ecs.html#ecsagent
    We provide the cluster name to the agent using the
    --cluster
    command line argument, and don't set it in
    ECSRun
    , and that works for us.
    Mike Vanbuskirk

    Mike Vanbuskirk

    2 months ago
    🤔
    I think the compositional structure of the Terraform module that Prefect created kind of makes labels redundant if you can’t use them to infer cluster
    is the
    --cluster
    option part of the
    prefect cli
    ?
    Mathijs Miermans

    Mathijs Miermans

    2 months ago
    Yeah, it an argument to agent ecs start: https://docs.prefect.io/api/latest/cli/agent.html#ecs-start
    Mike Vanbuskirk

    Mike Vanbuskirk

    2 months ago
    erm… huh, k… our agents are already running on the cluster… not sure where else they would run… I’m talking about flow jobs specifically
    seems like a data eng should not have to care about infra when writing flow jobs
    Mathijs Miermans

    Mathijs Miermans

    2 months ago
    It would be nice if the default behavior for the ECS agent was to start flows on the same ECS cluster as the agent was running on. I remember running into this early on.
    Mike Vanbuskirk

    Mike Vanbuskirk

    2 months ago
    agreed
    Anna Geller

    Anna Geller

    2 months ago
    if you need more ECS examples, check out this repo e.g. here https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows/ecs_flows_from_5411_pr you don't have to specify cluster name, e.g. here it will be inferred from the agent:
    RUN_CONFIG = ECSRun(
        labels=["ecs-agent-local"],
        task_definition_path="<s3://prefectdata/flow_task_definition.yaml>",
    )
    Mike Vanbuskirk

    Mike Vanbuskirk

    2 months ago
    awesome, thank you! re: inferring cluster, it may be an artifact of the TF config provided