Hi everyone — I’m struggling understanding how prefect behaves when I redeploy stuff. The way it works now is that 1 git repo = 1 prefect Project containing multiple flows. We have a dockerfile whose endpoint simply registers all the flows and then starts a LocalAgent. This is deployed as an ECS Service. When we deploy, we simply kill the existing service and start a new one with a new docker image — this means there is a little of “downtime” in terms of flow schedules and etc, like: • Time is 4.59 • We start a new deployment, meaning that at 4.59 the service is killed • There is a job scheduled to run at 5.00. • The service is only back up at 5.01. • This mean the 5.00 run will be fully missed. (please correct me if I’m wrong) My question, however, is about a second scenario, where we: • start run at 4.58, run takes about 5 minutes to finish, • We start new deployment at 4.59 — meaning there is a run in-progress • What happens with the flow run, if the entire service (local agent + code execution environment) is killed? • at 5.01 when the new deployment finishes, will prefect know to resume that flow run? how would it work? Reason I’m asking is because I plan on changing our deployment strategy to be blue/green, but I’m not sure how prefect will cope with a flow run being killed midway and etc Sorry if this is confusing! I appreciate any help.

@Bruno Murino you don't have to kill existing flow runs or agents when you deploy a new version of a flow. When your flow changes, you register a new version of it so that the next time you run it, you trigger a new version of the flow. This way you can entirely avoid any downtime related to new flow version deployment.

I’m struggling with understanding this because I’m using a LocalRun with a LocalAgent and LocalStorage — so the only way to change the flow code is by deploying a new docker image, and because I’m using a LocalAgent, this means I need the agent to be in that docker image as well

@Bruno Murino actually, with Local agent and storage, you don’t need to use Docker at all. You could deploy your agent e.g. in a virtual environment for isolation, but normally local agent is deployed as a local process without Docker. But there is also a Docker agent - perhaps this is what you’re looking for?

we need docker because we deploy as an ECS Service

Do you want your flows to run on ECS too? We have an ECS agent: https://docs.prefect.io/orchestration/agents/ecs.html I could share a tutorial on how to set it up if you’re interested

I have tried that but it was too slow to start the flow run

I see, I can actually totally understand that 🙂 the ECS with Fargate is really not the fastest option to start because the Serverless data plane needs to first provision the compute resources and pull the image before it can run the flow. But you could solve it by using EC2 instead of Fargate capacity provider.

we do use EC2 😂

Really? 😄 that’s weird. With EC2 data plane it should start up pretty much instantaneously. Could it be that the capacity provider was still set to Fargate?

don’t think so — we don’t have anything on fargate and I had to deal with our VPC and stuff, so it was definitely EC2

Anyway, in case you would want to go that path, you could use ECS CLI to spin up the cluster and then follow this or this to set up an agent as ECS service with the exception to change the capacity provider from FARGATE to EC2.

there might had been a problem with the ECS agent we had — it was deployed as an ECS Service and it would get an OOM error every 3 days or so, with no apaprent reason

hmm it never happened to me. Perhaps you could allocate more memory to the agent then?

yea I think I’ll give that a try

usually the agent doesn’t need a lot of memory because it doesn’t do any work by itself, it only spins up flows as ECS tasks

exactly, so I thought it was a bug at the time or something

yes, there is! There are two ways to store credentials that are retrieved by ECS tasks: 1. AWS Parameter Store 2. AWS Secrets Manager The blog post I linked handles that and includes a section on how to set it up using #1. You can also check out #3 in this blog https://aws.plainenglish.io/8-common-mistakes-when-using-aws-ecs-to-manage-containers-3943402e8e59?sk=334f367ff27d3fe9b56ff31f8b9ba447

I do use AWS secrets manager on ECS task definition that I create — but now Prefect is creating them and adding a bunch of environment variables

no no, Prefect will use the value from parameter or Secret - have a look at this section:

this is what I have setup:

but then you didn’t follow this tutorial, right? 😄 if you were, it would look like this - it must be an ARN to the Secret or Parameter

apologies — what tutorial are you referring to? the link you sent doesn’t mention Prefect at any point

I mentioned it here: https://prefect-community.slack.com/archives/CL09KU1K7/p1638442488159300?thread_ts=1638438180.148200&cid=CL09KU1K7 This is the ECS Agent walkthrough: https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df

on the link you sent the task definition is created for the ECS Agent — which is fine — but my issue with on the task definition for flows

nice! so it looks like you git it working? LMK if you have any open questions.

well not really! haha sorry for the confusion

@Bruno Murino I see. The only way this may happen is if your ECSRun is using a different task definition than your agent. If you don’t set it explicitly, the one from the agent will be used. I’ve recently updated the docstring in the ECSRun to include more examples - perhaps you can reference the same task definition ARN as the one used by your agent?

I’m afraid the task run of the flow has to be different from the task definition of the agent — mainly because they run different images

network mode belongs to the task definition, but the exact network details such as VPC id and subnet id belong to run task

well I need “networkMode = bridge” anyway haha

true, for those you don’t need VPC details, correct

do you mind commenting on my assumption “I suspect what Prefect does is a “container overrides” to inject all envs vars it requires — and that bubbles up to the AWS console”

correct, if you set something explicitly on ECSRun, it will serve as overrides to what was configured on the agent. You can see more here: https://github.com/PrefectHQ/prefect/blob/d44b72a950ebda9f7bc6a9712fc71e2e9c680d25/src/prefect/agent/ecs/agent.py#L444-L475

Nice! Thanks for showing me! From what I gather, if I set a container overrides on some env vars it will get ignored, however since the overrides I want to place are “secrets” instead of “environment”, it should still be applied. The bit I need to test is if some env var is in both the “environment” section and the “secrets” section, then which way takes precedence