Hi Prefect community! I’m using prefect cloud and deploy flow runs using the ECS Fargate agent successfully. To trigger a flow run I need to first be running a

prefect ecs agent

in a terminal, and then use the prefect dashboard to trigger that run. I’d like to build a standalone app, so have a custom GUI that lets a user input flow parameters and trigger a flow run. I’m imagining that I’ll have to create a prefect ecs agent using the prefect api instead of running it in a separate terminal but I’m not sure how to go about that. Pls let me know of any deployment recipes that fit my use case! 🙏

You need to have an agent running somewhere at any time that you want a flow to run. Typically you have them running in the cloud.

Thank you @Spencer ! My question is more about what are the steps needed to set up an agent without having to do the manual

open terminal->run prefect ecs agent start

. Let’s say I have a file called

run_flows.py

, what would the set up of an agent look like in that file? I hope this makes it more clear!

The flow running script wouldn't setup an agent. The agent would, typically, just be running all the time separately. The

run_flows.py

script would call the Prefect API with the desired

FlowRun

parameters and the agent would eventually pick up those parameters from the API.

Hi @Fina Silva-Santisteban - as @Spencer mentioned, you'd typically stand up an agent as a standalone process. This can mean a few different things depending on your desired infrastructure; the most straightforward way to accomplish that would be to spin up a light-weight machine (a small EC2 instance, for example) and nohup an agent on that machine. Some users (including us at Prefect!) bake agent spinup into CI/CD pipelines so that agents are 1) up to date and 2) always deployed with new infrastructure changes.

Thank you @nicholas! Can you pls share the steps needed to spinup an agent into a CI/CD pipeline? Would it also be possible to run it in an ECS Fargate Cluster? (Easier to maintain than my own EC2 instance)

The steps would be the same as spinning up an agent locally; you would bake the agent startup command into your pipeline as you would with any other step! And certainly you could run an agent as part of a fargate cluster - the setup would be identical to defining an application in a container on the cluster, only in this case the application would be a prefect agent 🙂

Hello @Fina Silva-Santisteban! I believe ECS provides services for long running tasks. Here you can read how it works https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html. But basically you need to provide a task definition for your agent and run it as a service, instead of spinning EC2

Thank you @nicholas and @Mariia Kerimova! I’m a bit fuzzy on how to set up the Task. The Task definition requires an image to run the container. Do I need to first upload a copy of the prefect docker image into one of my ECS repositories? Or is there a way to pull that one directly for the Task?

Hi @Fina Silva-Santisteban - if you're going to use an ECS registry you can create an image there that is built on top of the prefect image, i.e.:

FROM prefecthq/prefect:latest

# other commands you want to run

Thanks so much @nicholas! When I check the prefect docker hub registry I see some images with version numbers, and some with both version numbers and either

core

or

python

attached to the name as well. Since I’m using prefect 0.14.5 locally it seems like that version would make the most sense for us, but there are 5 different images with that version number. Which one would you suggest and why? 🤔

The tags with

python

are built with a specific version of python installed (the version is also in the tag) - use those if you need to pin to a specific python version for your code. Otherwise you could use the

core

images to get the utils you'll need for your flows (without things like the CLI or Server)

Thank you @nicholas ! I know which one to use then! Thank you everyone for your help!