Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello! prefect newbie here!
our company is considering implementing prefect to our existing data pipeline process and was wondering
Is it possible to configure prefect with multiple ec2 instances?
we have different servers with ml training job, and batch job and would like to use one prefect to manage all tasks in different instances
I saw the concepts of ‘agent’ so in this case would it be producing agent in each server and connecting them to master node?
Thanks in advance!

image.png

What is your existing infra for managing this instance? Have you tried something like Amazon ECS:

I’m also new to AWS but I assume once you have a ECS set up, you can use Prefect EC2 Agent to talk to the ECS.

And then you’ll orchestrate your tasks in multiple EC2 servers

Is this sounds like a promising direction? Let me know how you think :cool-llama:

wow it sounds great! I will definitely take a look into the solution with ecs, but one of my concern is that one of our ml training server is not part of the aws ecs, its on premise server. So I’m wondering if there is a way to orchestrate all in one! thanks for your insight though :smile: <@U025W1XT18V>

Screen Shot 2021-07-27 at 4.53.03 PM.png

I see, so the objective is really about how to integrate flow information across your instance

In that case, how about creating a `ML-project` that contains all the flows together?

In the screenshot here you can see two flows are registered under the same project.

You might need to create an agent for EC2 and agent  for all the on-perm servers ( cc <@U01QEJ9PP53> to confirm this )

but I just want to point out here, that with Prefect UI, it’s not so much of a problem of having multiple flows running across servers. The dashboard and the `project / flow` structure allows you have a glance of your system as a whole, but also down to the detail of a particular flow

sweet! now I have an abstract idea of how to go about thank you so much it really helped alot :smile: <@U025W1XT18V>

Hey <@U0299BKCMKM>, I think the question here is how to management different compute with one Prefect backend (Cloud or Server). Prefect was designed for this. You can have multiple agents in separate infrastructure. Prefect uses <https://docs.prefect.io/orchestration/execution/overview.html#labels|labels> to identify where a flow should be run.

You can have an agent for EC2 and an agent for your on prem infrastructure. Maybe you can have an agent for compute with GPU. Agents are lightweight long-running process that poll the backend to see if there are any flows to execute.

Prefect Cloud (or Server) will handle the orchestration for several agents, and there is the Agents tab in the UI that lets you keep track of them. <@U025W1XT18V>’s suggestion of a project for all relevant flows is also good as you get the timeline of the flow runs in that project