https://prefect.io logo
f

Fina Silva-Santisteban

02/18/2021, 1:27 AM
Hi Prefect community! I’m using prefect cloud and deploy flow runs using the ECS Fargate agent successfully. To trigger a flow run I need to first be running a
prefect ecs agent
in a terminal, and then use the prefect dashboard to trigger that run. I’d like to build a standalone app, so have a custom GUI that lets a user input flow parameters and trigger a flow run. I’m imagining that I’ll have to create a prefect ecs agent using the prefect api instead of running it in a separate terminal but I’m not sure how to go about that. Pls let me know of any deployment recipes that fit my use case! 🙏
s

Spencer

02/18/2021, 1:34 AM
You need to have an agent running somewhere at any time that you want a flow to run. Typically you have them running in the cloud.
f

Fina Silva-Santisteban

02/18/2021, 1:37 AM
Thank you @Spencer ! My question is more about what are the steps needed to set up an agent without having to do the manual
open terminal->run prefect ecs agent start
. Let’s say I have a file called
run_flows.py
, what would the set up of an agent look like in that file? I hope this makes it more clear!
s

Spencer

02/18/2021, 1:39 AM
The flow running script wouldn't setup an agent. The agent would, typically, just be running all the time separately. The
run_flows.py
script would call the Prefect API with the desired
FlowRun
parameters and the agent would eventually pick up those parameters from the API.
The agent is meant to pick up scheduled flows from the Prefect scheduler. If you aren't using any schedules and just running a flow directly, you can always just call
flow.run()
on the
Flow
object itself rather than going through the scheduler/API.
n

nicholas

02/18/2021, 1:42 AM
Hi @Fina Silva-Santisteban - as @Spencer mentioned, you'd typically stand up an agent as a standalone process. This can mean a few different things depending on your desired infrastructure; the most straightforward way to accomplish that would be to spin up a light-weight machine (a small EC2 instance, for example) and nohup an agent on that machine. Some users (including us at Prefect!) bake agent spinup into CI/CD pipelines so that agents are 1) up to date and 2) always deployed with new infrastructure changes.
f

Fina Silva-Santisteban

02/18/2021, 1:46 AM
Thank you @nicholas! Can you pls share the steps needed to spinup an agent into a CI/CD pipeline? Would it also be possible to run it in an ECS Fargate Cluster? (Easier to maintain than my own EC2 instance)
n

nicholas

02/18/2021, 1:50 AM
The steps would be the same as spinning up an agent locally; you would bake the agent startup command into your pipeline as you would with any other step! And certainly you could run an agent as part of a fargate cluster - the setup would be identical to defining an application in a container on the cluster, only in this case the application would be a prefect agent 🙂
Think of the Prefect agent as nothing more than a standalone process that can be run anywhere, so long as it can make requests to Prefect Cloud (or Server) and submit work to your chosen execution environment.
m

Mariia Kerimova

02/18/2021, 3:11 PM
Hello @Fina Silva-Santisteban! I believe ECS provides services for long running tasks. Here you can read how it works https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html. But basically you need to provide a task definition for your agent and run it as a service, instead of spinning EC2
f

Fina Silva-Santisteban

02/18/2021, 4:56 PM
Thank you @nicholas and @Mariia Kerimova! I’m a bit fuzzy on how to set up the Task. The Task definition requires an image to run the container. Do I need to first upload a copy of the prefect docker image into one of my ECS repositories? Or is there a way to pull that one directly for the Task?
n

nicholas

02/19/2021, 4:13 PM
Hi @Fina Silva-Santisteban - if you're going to use an ECS registry you can create an image there that is built on top of the prefect image, i.e.:
Copy code
FROM prefecthq/prefect:latest

# other commands you want to run
🙏 1
Though I'd recommend pinning to a specific image tag, rather than using latest
f

Fina Silva-Santisteban

02/19/2021, 4:59 PM
Thanks so much @nicholas! When I check the prefect docker hub registry I see some images with version numbers, and some with both version numbers and either
core
or
python
attached to the name as well. Since I’m using prefect 0.14.5 locally it seems like that version would make the most sense for us, but there are 5 different images with that version number. Which one would you suggest and why? 🤔
n

nicholas

02/19/2021, 5:04 PM
The tags with
python
are built with a specific version of python installed (the version is also in the tag) - use those if you need to pin to a specific python version for your code. Otherwise you could use the
core
images to get the utils you'll need for your flows (without things like the CLI or Server)
f

Fina Silva-Santisteban

02/19/2021, 5:22 PM
Thank you @nicholas ! I know which one to use then! Thank you everyone for your help!
🚀 1