Sean Talia
01/28/2021, 5:30 PMtask_definition_arn
, task_definition_path
, task_definition
) to the flow considered to be canonical? i'm trying to weigh the pros and cons of having some other process like terraform handle management of the task definition (thereby only leaving the ARN to be supplied to prefect) vs. using prefect to handle thatrevision
being out of sync with what your prefect flow has in its definition?Jim Crist-Harif
01/28/2021, 6:02 PMtask_definition
if possible (it's the simplest for you to manage). task_definition_path
is good if you don't want to store the task definition in prefect cloud's database, task_definition_arn
is good if you want to control when and how the task definition is registered (and lets you avoid giving the agent permission to register tasks). If you don't have super strict security requirements though, task_definition
is probably the most user friendly.Sean Talia
01/28/2021, 6:50 PMECSRun
flow? this:
ECS Tasks are composed of task definitions and runtime parameters.
makes it sound like:
flow.run_config = ECSRun(
task_definition="<s3://bucket/path/to/task.yaml>",
image="example/my-custom-image:latest",
cpu="2 vcpu",
)
Jim Crist-Harif
01/28/2021, 7:01 PMECSRun
(and all run-configs) are for specifying where your flow process runs. Your flow code itself is stored in your specified flow.storage
, and contains the DAG of your flow.Sean Talia
01/28/2021, 7:08 PMJim Crist-Harif
01/28/2021, 7:18 PMECSRun
run config (and all run configs) are for specifying how to setup and deploy a process for running a prefect flow. The task definition used by an ECSRun
must be used for running a flow run, not some arbitrary task. If you're trying to use prefect to manage arbitrary ECS tasks, you'll need to write the logic for starting (or registering or monitoring or ...) as prefect tasks and encode them into a larger flow. For example, you might write a prefect flow that runs in a single ECS task (described by ECSRun
) that kicks off and monitors other arbitrary ECS tasks that you wrote yourself.Sean Talia
01/28/2021, 7:28 PM