I am building a deploy with github and docker. I tested locally and it works. My question is, Can i build the deploy locally on my machine and then will it run where ever I put the agent or does the deploy need to be made wherever its going to run? (probably sounds really stupid to someone that knows what they are doing 😅 )

Hey @Adam - containerized orchestration is complicated, all questions are welcome 🙂 I believe I remember you're on prefect 2, so I'll speak in those terms when its time for your deployment to run, your agent will pull your storage and infrastructure blocks from Cloud (or your hosted server) down to where its running (like a VM, or a kubernetes pod), which will answer the agent's question: "How should I submit this flow run?" for docker, if your agent gets a DockerContainer infra block, it will assume it has a docker environment to submit to, for example, if I run docker desktop on my local and open a terminal and run

prefect agent start

, an agent would attempt to submit flow runs to docker desktop (pulling the

image

specified on your infra block) so for example, you could install docker on a vm, set

PREFECT_API

env vars, and do

prefect agent start

(prob via something like systemd) and you could start running containerized flow runs on that VM just a side note that if you're a k8s user, the helm chart is the easiest way to get going very quickly

awesome! super helpful! One more question... been talking to eng at my company and they mentioned using ecs tasks to run my orchestrations. I found some articles, this one in particular. I am curious if ECS is in place of docker, as an Infra block or can be used together. The eng team was looking at using a spin up instance as opposed to always on. Maybe helm answers that as I have not read it yet. I was getting confused as docker deploy would need an agent up to run. Felt somewhat chicken and egg in my brain that deploys need an agent and wasnt sure how one gets triggered to spin up just in time, if that makes sense.

@Adam In my situation I run deployments on the server where my agent will run. But that is because in my deployments I have some environment variables that it pulls from the code environment to properly deploy. Then when the agent picks up the flow run it has the proper filled in values for it’s environment.

appreciate the help again! been asking a lot lately

anytime!

ECSTask

is a different type of infra block which would replace

DockerContainer

, although they are similarly rooted in containerization 😄 (as in, you could

docker build / push

your image to ECR in a way that doesnt involve Prefect's

DockerContainer

at all, and then use

ECSTask

infra block) helm is for those who bring their own k8s cluster (managed or otherwise), which doesn't sound like your situation if you're looking at ECS regardless of eventual runtime infra, you'll need an agent sitting somewhere constantly asking "do I have any work to submit?". in the case of ECS, that can pretty much be anywhere if you attach your

AWSCredentials

to the

ECSTask

block

so it seems like to use prefects blocks I have to use docker OR ECS and then potentially custom build the other if thats what we go with

custom build the other if thats what we go with

hmm I'm not certain what you mean but just to reiterate in case thats helpful, basically each deployment (at this time) needs a single infra block, which can be any of the valid infra block types (including DockerContainer and ECSTask) - multiple of which are docker-based in nature. in any docker-based runtime, you'll need to build a custom image if you have custom dependencies outside of the prefect base image - for ECS, it makes sense to push your custom image to ECR, but its totally up to you / your objectives

I meant if I can only have one Infra block I have to choose between ECS and docker block. This makes sense! thanks for the help

sure thing!

I also see what you are saying with custom image and its kinda what I was saying too lol, use the ECS block and make a custom image file instead of using the docker block and figure out how to deploy ECS

@Nate A couple of follow up questions, on the helm agent and ECS, here - https://prefect-community.slack.com/archives/CL09KU1K7/p1683785697841039 Would you be so kind to advise please? Thanks a ton in advance 🙏