I have a setup in which I provision my infrastructure via AWS CDK. I run a worker on ECS. The same ECS task is responsible for running flow runs. Upon a new code change, my CI/CD pipeline updates the ECS task, thereby killing any active Prefect flow runs. How can I avoid this? I’d like to keep my code in sync with my infrastructure stack and not have to kill any running Prefect jobs, but this sync necessitates a task restart afaik.

Hey are you running a ProcessWorker on ECS? Could you help me understand why your CI/CD pipeline needs to update the ECS task when you push new code?

Yes, running a Process worker. When we merge new code, we package a new image and in order to deploy that new image, you run

aws ecs update-service

which triggers a rolling task restart

I believe what you're saying is: Your image contains your code + prefect. And that image is deployed where you start a process worker from it. When running flows the code is already present on the image

That’s right

You can do a couple different things, mainly you should be looking to de-couple your code from the worker. You should be able to run a worker from the prefect image and then you have a couple options: 1. Process Worker + pulling your code from a remote location like s3/github in the

deployment.pull_steps

2. Image Based Worker (docker, ecs, etc.) a. have your ci/cd pipeline build and update the image b. update your work/pool deployment to use your image c. then your worker will be deploying your image each time for each flow run (for example for ECSWorker it will deploy a new ECS task for each flow run)

Yes it does! Is there a guide for option 2 that I can take a look at? I was looking here and here but the former seems like an outdated guide based on ECSTask and the latter seems like it only accomplishes the first half of this

I would have linked the 2nd guide. Could explain a little more what seems to be missing and I can try to answer?

Sure, maybe I am misunderstanding but it seems to me that by the end of Step 2 I have a work pool which pushes work to an ECS service. The part that is missing is how that ECS service farms the task run to another ECS task.

Got it. The command the worker is running from is

prefect worker start --pool "my-ecs-pool"

If the work pool is of type

ecs

, this command will dynamically spin up an

ECSWorker

(as opposed to a process work pool, will lead to a

ProcessWorker

being started) An

ECSWorker

is typed "ECS" because it reads from ECS works pools and DEPLOYS flow runs as ECS tasks. The Worker is just a process that can run anywhere, you could run it off your laptop for example. The worker as an ECS Service is likely just convenient

Gotcha! I’m gonna play around with this setup then. Cheers.