https://prefect.io logo
Channels
Powered by
s

Sean Talia

01/28/2021, 5:30 PM
when using the ECS runconfig, is one of the 3 methods for attaching the ecs task (
task_definition_arn
, 
task_definition_path
, 
task_definition
) to the flow considered to be canonical? i'm trying to weigh the pros and cons of having some other process like terraform handle management of the task definition (thereby only leaving the ARN to be supplied to prefect) vs. using prefect to handle that
it seems like using prefect to manage it is the better move because you don't need to worry about the task's 
revision
being out of sync with what your prefect flow has in its definition?
j

Jim Crist-Harif

01/28/2021, 6:02 PM
Yeah, I'd recommend using 
task_definition
if possible (it's the simplest for you to manage). 
task_definition_path
is good if you don't want to store the task definition in prefect cloud's database, 
task_definition_arn
is good if you want to control when and how the task definition is registered (and lets you avoid giving the agent permission to register tasks). If you don't have super strict security requirements though, 
task_definition
is probably the most user friendly.
s

Sean Talia

01/28/2021, 6:50 PM
okay cool that all makes sense
I guess another question I have though is, is there anything at all that would go into the flow body/context for an 
ECSRun
flow? this:
Copy code
ECS Tasks are composed of task definitions and runtime parameters.
makes it sound like:
Copy code
flow.run_config = ECSRun(
    task_definition="<s3://bucket/path/to/task.yaml>",
    image="example/my-custom-image:latest",
    cpu="2 vcpu",
)
is totally sufficient to run the flow
do these flows end up looking like, empty DAGs? or do they build out some schematic under the hood that you can only see once you've registered the flow
j

Jim Crist-Harif

01/28/2021, 7:01 PM
I'm not sure I follow the question. 
ECSRun
(and all run-configs) are for specifying where your flow process runs. Your flow code itself is stored in your specified 
flow.storage
, and contains the DAG of your flow.
Perhaps the confusion is the "Task" part of "ECS Task"? This is referring to AWS's terminology (ECS "jobs" are "tasks" in AWS's lingo). This has nothing to do with Prefect "tasks".
s

Sean Talia

01/28/2021, 7:08 PM
mmm i might just be confusing myself, i think need to get a little more hands on with the ECS stuf
just at a high level though I was thinking, if I have an ECS Task already registered to AWS that will use an image that already has a pre-defined entrypoint/cmd that will execute whatever i need it to execute, then there's not necessarily anything further that I need prefect to "do", as launching the ECS Task already accomplishes what I want; I don't need to write any additional prefect tasks into the flow body
if that doesn't make any sense at all feel free to ignore me 🙂
j

Jim Crist-Harif

01/28/2021, 7:18 PM
Ah, gotcha. The 
ECSRun
run config (and all run configs) are for specifying how to setup and deploy a process for running a prefect flow. The task definition used by an 
ECSRun
must be used for running a flow run, not some arbitrary task. If you're trying to use prefect to manage arbitrary ECS tasks, you'll need to write the logic for starting (or registering or monitoring or ...) as prefect tasks and encode them into a larger flow. For example, you might write a prefect flow that runs in a single ECS task (described by 
ECSRun
) that kicks off and monitors other arbitrary ECS tasks that you wrote yourself.
👍 1
s

Sean Talia

01/28/2021, 7:28 PM
alright there we go, that's the insight i was lacking, thank you!
Powered by