Sean Talia
02/22/2021, 9:53 PMregister_task_definition()
method. Does this mean that even if the contents of task definition hasn't actually changed, (e.g. the flow is never re-registered, or it's registered but with an identical task definition), the Agent is still going to register a new task to AWS on each and every flow run?Kyle Moon-Wright
02/22/2021, 10:55 PMtask_definition_arn
kwarg, otherwise Prefect will define this task for you upon registration.
FYI I’m trying to have this verified, so I’ll update you once I know more.Sean Talia
02/22/2021, 11:45 PMrevision
count for the task (unless Amazon is doing some kind of hashing on their end and sees that in fact, nothing changed – maybe that is the case?)...is there a specific reason why the implementation was done this way rather than, say, registering the task with AWS at the same time that the Flow is registered to Prefect (e.g. more or less what happens when S3 is used for storage), caching the task ARN as a part of the Flow's metadata, and then just using that task ARN at Flow run time?Kyle Moon-Wright
02/23/2021, 12:40 AMrun_config
allows for those idempotency concerns. In general, Prefect doesn’t make any assumptions about how you’ve configured your infrastructure and accommodates a wide range of setups, but also can’t reach into your AWS infra to see what fits best, so it delegates each flow run to defaults in order of precedence (the run_config will override the Agent). If you’d prefer all flow runs sent to that Agent to have that definition, you can use the task_definition_path
on the Agent.
Therefore, if you reregister your flow with an ECSRun
and a task_definition_arn
, then your Task will be replaced with its updated version. However, if you reregister your flow without this defined and submit it to your Agent without a definition path/arn anywhere, Prefect can’t assume it already exists and reverts to creating a new Task definition.