https://prefect.io logo
Title
a

Adam

03/27/2023, 4:33 PM
I am building a deploy with github and docker. I tested locally and it works. My question is, Can i build the deploy locally on my machine and then will it run where ever I put the agent or does the deploy need to be made wherever its going to run? (probably sounds really stupid to someone that knows what they are doing πŸ˜… )
n

Nate

03/27/2023, 6:07 PM
Hey @Adam - containerized orchestration is complicated, all questions are welcome πŸ™‚ I believe I remember you're on prefect 2, so I'll speak in those terms when its time for your deployment to run, your agent will pull your storage and infrastructure blocks from Cloud (or your hosted server) down to where its running (like a VM, or a kubernetes pod), which will answer the agent's question: "How should I submit this flow run?" for docker, if your agent gets a DockerContainer infra block, it will assume it has a docker environment to submit to, for example, if I run docker desktop on my local and open a terminal and run
prefect agent start
, an agent would attempt to submit flow runs to docker desktop (pulling the
image
specified on your infra block) so for example, you could install docker on a vm, set
PREFECT_API
env vars, and do
prefect agent start
(prob via something like systemd) and you could start running containerized flow runs on that VM just a side note that if you're a k8s user, the helm chart is the easiest way to get going very quickly
a

Adam

03/27/2023, 6:14 PM
awesome! super helpful! One more question... been talking to eng at my company and they mentioned using ecs tasks to run my orchestrations. I found some articles, this one in particular. I am curious if ECS is in place of docker, as an Infra block or can be used together. The eng team was looking at using a spin up instance as opposed to always on. Maybe helm answers that as I have not read it yet. I was getting confused as docker deploy would need an agent up to run. Felt somewhat chicken and egg in my brain that deploys need an agent and wasnt sure how one gets triggered to spin up just in time, if that makes sense.
appreciate the help again! been asking a lot lately
j

Jarvis Stubblefield

03/27/2023, 6:19 PM
@Adam In my situation I run deployments on the server where my agent will run. But that is because in my deployments I have some environment variables that it pulls from the code environment to properly deploy. Then when the agent picks up the flow run it has the proper filled in values for it’s environment.
n

Nate

03/27/2023, 6:21 PM
appreciate the help again! been asking a lot lately
anytime!
ECSTask
is a different type of infra block which would replace
DockerContainer
, although they are similarly rooted in containerization πŸ˜„ (as in, you could
docker build / push
your image to ECR in a way that doesnt involve Prefect's
DockerContainer
at all, and then use
ECSTask
infra block) helm is for those who bring their own k8s cluster (managed or otherwise), which doesn't sound like your situation if you're looking at ECS regardless of eventual runtime infra, you'll need an agent sitting somewhere constantly asking "do I have any work to submit?". in the case of ECS, that can pretty much be anywhere if you attach your
AWSCredentials
to the
ECSTask
block
a

Adam

03/27/2023, 6:59 PM
so it seems like to use prefects blocks I have to use docker OR ECS and then potentially custom build the other if thats what we go with
n

Nate

03/27/2023, 7:34 PM
custom build the other if thats what we go with
hmm I'm not certain what you mean but just to reiterate in case thats helpful, basically each deployment (at this time) needs a single infra block, which can be any of the valid infra block types (including DockerContainer and ECSTask) - multiple of which are docker-based in nature. in any docker-based runtime, you'll need to build a custom image if you have custom dependencies outside of the prefect base image - for ECS, it makes sense to push your custom image to ECR, but its totally up to you / your objectives
a

Adam

03/27/2023, 8:52 PM
I meant if I can only have one Infra block I have to choose between ECS and docker block. This makes sense! thanks for the help
n

Nate

03/27/2023, 8:54 PM
sure thing!
a

Adam

03/27/2023, 9:23 PM
I also see what you are saying with custom image and its kinda what I was saying too lol, use the ECS block and make a custom image file instead of using the docker block and figure out how to deploy ECS
r

Ritabrata Moitra

05/11/2023, 6:16 AM
@Nate A couple of follow up questions, on the helm agent and ECS, here - https://prefect-community.slack.com/archives/CL09KU1K7/p1683785697841039 Would you be so kind to advise please? Thanks a ton in advance πŸ™