https://prefect.io logo
c

Charles Lariviere

12/07/2020, 3:52 PM
Hey 👋 I’m trying to figure out how to properly deploy Prefect on AWS ECS/Fargate such that we always have an agent running, ready to pick up any scheduled flow. I was able to execute
prefect agent fargate start -t $PREFECT_TOKEN
locally which appeared to work (it successfully started and was waiting for flows), but; 1. It never received any flows triggered from Prefect Cloud 2. This seems to be tied to my local terminal session (i.e. if I close my terminal, the agent shuts down) 3. I can’t see where this is actually deployed — I was expecting a new cluster to be created in AWS ECS, but that’s not the case Am I misunderstanding how this is supposed to work? If so, how can we go about deploying an always-running agent that’s ready to deploy new Fargate tasks whenever a flow is scheduled? Or is this not how Prefect is supposed to work either?
j

Jim Crist-Harif

12/07/2020, 4:32 PM
Hi Charles, I'm sorry you're having issues with this. A few quick notes:
It never received any flows triggered from Prefect Cloud
I'd check the labels configured on the agent/flows. These need to match (agents filter on labels) or your agent won't pick up any flow runs. See https://docs.prefect.io/orchestration/agents/overview.html#flow-affinity-labels for more info.
This seems to be tied to my local terminal session (i.e. if I close my terminal, the agent shuts down)
Yes, running
prefect agent <...> start
will start an agent process locally. Since many users have different tools they like to use for managing long running processes (k8s, docker, supervisord, cloud tools, etc...) we're agnostic to how you manage to keep the agent process running. Note that agents are stateless - killing an agent doesn't terminate any of the jobs started by it - you only need to have an agent process running when you want new jobs to be started.
I can’t see where this is actually deployed — I was expecting a new cluster to be created in AWS ECS, but that’s not the case
It will use your default cluster if one isn't specified.
I'm in the process of rewriting our agent docs (hopefully done late today/tomorrow), hopefully the updated docs will help answer your confusion here.
Also, note that we're in the process of deprecating the fargate agent in favor of the ECS agent (
prefect agent ecs start
). Besides api docs (https://docs.prefect.io/api/latest/cli/agent.html) there's nothing I can point you at (yet) for that. Docs coming soon! The fargate agent will stick around for a few months though, so no rush to change, but it won't support the new run-config pattern (https://docs.prefect.io/orchestration/flow_config/run_configs.html) that we've moved to.
c

Charles Lariviere

12/07/2020, 5:37 PM
Ok interesting! Looking forward to the ECS docs — this is likely how we’ll deploy.
you only need to have an agent process running when you want new jobs to be started.
If I have a flow that has a schedule to run daily at 02:00 in Prefect Cloud; my understanding is that I would need to either have an agent always running in ECS to receive the flow run daily at 02:00 and execute it — or can Prefect Cloud also start an agent? If
prefect agent ecs start
is tied to a terminal session, I don’t understand how I could have an agent that’s always running on ECS with this command. Either I run this command locally to spin up and ECS agent that’s tied to my local terminal session, or I manually deploy a long-running task in ECS and run the command
prefect agent local
within the ECS task.
j

Jim Crist-Harif

12/07/2020, 5:59 PM
can Prefect Cloud also start an agent?
Agents run on your infrastructure, you'd need to have one running to accept and start the flow run
I don’t understand how I could have an agent that’s always running on ECS with this command.
How you start a long running agent is up to you, there's too many different ways people manage infrastructure for us to support them all with quick CLI commands. In your case, you might start a long running ECS task that runs
prefect agent ecs start
, and has appropriate permissions to kick off other ECS tasks.
m

Marc Lipoff

12/11/2020, 3:52 PM
Hi @Jim Crist-Harif I want to follow up on this. I'm a bit confused on the "best" way to deploy prefect agents to ECS, as well. Do you have any examples of how to use terraform to deploying agents? Any examples/documentation you have on how to in-practice deploy an ecs cluster (and agents) would be great. Thanks
n

nicholas

12/11/2020, 4:55 PM
Hi @Marc Lipoff - I don't have any specific examples of using terraform to deploy agents but creating a long running agent with terraform shouldn't be different from starting a machine with an image and invoking a remote-exec command. So something like this:
Copy code
resource "aws_instance" "web" {
  # the rest of your machine setup

  provisioner "remote-exec" {
    command = "prefect agent ecs start"
  }
}
Note that the only traffic you'd need to allow through your security group is outbound - Prefect doesn't communicate back to the agent.
3 Views