Sean Talia02/22/2021, 9:53 PM
method. Does this mean that even if the contents of task definition hasn't actually changed, (e.g. the flow is never re-registered, or it's registered but with an identical task definition), the Agent is still going to register a new task to AWS on each and every flow run?
Kyle Moon-Wright02/22/2021, 10:55 PM
kwarg, otherwise Prefect will define this task for you upon registration. FYI I’m trying to have this verified, so I’ll update you once I know more.
Sean Talia02/22/2021, 11:45 PM
count for the task (unless Amazon is doing some kind of hashing on their end and sees that in fact, nothing changed – maybe that is the case?)...is there a specific reason why the implementation was done this way rather than, say, registering the task with AWS at the same time that the Flow is registered to Prefect (e.g. more or less what happens when S3 is used for storage), caching the task ARN as a part of the Flow's metadata, and then just using that task ARN at Flow run time?
Kyle Moon-Wright02/23/2021, 12:40 AM
allows for those idempotency concerns. In general, Prefect doesn’t make any assumptions about how you’ve configured your infrastructure and accommodates a wide range of setups, but also can’t reach into your AWS infra to see what fits best, so it delegates each flow run to defaults in order of precedence (the run_config will override the Agent). If you’d prefer all flow runs sent to that Agent to have that definition, you can use the
on the Agent. Therefore, if you reregister your flow with an
, then your Task will be replaced with its updated version. However, if you reregister your flow without this defined and submit it to your Agent without a definition path/arn anywhere, Prefect can’t assume it already exists and reverts to creating a new Task definition.