Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi there! I'm working on understanding Prefect Cloud infrastructure. I'm coming from an Airflow background. I'm trying to understand the difference between orchestration infrastructure and task execution infrastructure when using Prefect Cloud. Prefect Cloud manages the orchestration infrastructure for you, but does it also manage the task execution infrastructure or do you have to set that up separately? If we wanted to, for example, run tasks in a docker container or as non-containerized python scripts.

We're not really interested in running anything locally and interested in a full cloud setup, but the documentation seems to assume a hybrid model where things are run locally and I'm having trouble understanding how to set up a full cloud-based model, and whether we'll need anything beyond what is covered in Prefect Cloud.

I think my understanding is that we'd need to run an agent in our own cloud infrastructure to accomplish this, and that Prefect Cloud does not run any execution infrastructure, but want to confirm.

That's my understanding as well - Containers, VMs, etc.

I'm trying to do a test with an agent on EC2 right now.

That's right <@U04LLKP8KEC>, agents are essentially dispatchers that live in your cloud env and make the bridge between Prefect Cloud and your internal infrastructure. Prefect Cloud only manages the flow queue and stores the data (logs, task and flow statuses etc) that the agent sends it.

Agents poll the queue using the Prefect REST API (which is great, as you don't need to give Prefect Cloud any kind of access to your cloud env for that to happen), pull pending flow run requests, then dispatch those requests to be run in a job runner. Jobs can be run in the agent's VM/container itself but the agent can also create, for instance, Kubernetes jobs to run those for you. That's the setup we've been using here actually.

We also run the agents on Kubernetes itself.