https://prefect.io logo
Title
j

Jacob Bedard

02/23/2023, 6:15 PM
I'm having trouble wrapping my head around all these various blocks, and there isn't a lot of great content out there, in my opinion. Everyone who does the prefect 2 videos sounds like they're very close to the product, and it's honestly a bit off-putting because they don't use any terminology from prefect cloud v1. Am I correct in understanding a setup could work like this: • I set up a repo for my prefect flows • I tie that repo to an S3 bucket or other storage location so it receives all updates and new files from the master/prod branch • I save a reference to this storage location as a prefect block • when I make a new flow, I can still test locally (I'm clear on this part at least), then when I'm ready to push to production, I push my code to my repo ◦ my repo updates the code storage location (S3 in my example) • I then create a deployment, and in the deployment, I state which named flow function is to be used from which file within the storage location, and I do this using a prefect "storage block" • the prefect cloud will then tell my agent (let's do a simple example and say I have an EC2 worker) that there is a new job to run • the worker, using the storage location specified in the deployment, will reach into the s3 bucket, and execute that flow
r

redsquare

02/23/2023, 7:06 PM
The flow and associated files are uploaded to the s3 block on deployment
the block is passed into the deployment essentially
your git repo is not connected to the s3 bucket in anyway
:upvote: 1
j

Jacob Bedard

02/23/2023, 7:11 PM
So, the actual code is being sent when I specify a storage block?
Do you use CI/CD?
I'm trying to work that in here somehow too
r

redsquare

02/23/2023, 7:13 PM
yes I use github actions which deploy the new/changed flows on push
n

Nate

02/23/2023, 7:14 PM
Hi @Jacob Bedard feel free to let us know what type of specific content you're looking for, and we can either direct you to that or explore creating more of it. I think your general idea of a setup is mostly there - i would just add a couple clarifying points: • as redsqaure alludes to, when you create a deployment, it essentially asks you three questions ◦ which flow do you want to deploy? you answer with an
entrypoint
(i.e.
localfile.py:flow_decorated_fn
) ◦ where should this flow get uploaded (and pulled from at runtime)? you answer with a storage block (e.g.
s3/mybucket/mysubpath
) ◦ how should this flow run? you answer with an infra block (e.g.
kubernetes-job/my-k8s-job
) • code is uploaded to location specified by your storage block (if possible) when you run
prefect deployment build ...
(and incorporate the info above)
j

Jacob Bedard

02/23/2023, 7:15 PM
Ok, this is the part I think I was missing - the code actually being sent to the specified storage block
r

redsquare

02/23/2023, 7:15 PM
j

Jacob Bedard

02/23/2023, 7:16 PM
Thanks to both of you for clarifying that!
b

Bianca Hoch

02/23/2023, 7:16 PM
Just to add my two cents as well, @Jacob Bedard if you haven't already, check out our recipes. Particularly Deployments & CICD and AWS Infra.
j

Jacob Bedard

02/23/2023, 7:16 PM
I thought the code had to be in the storage block, because the whole thing with prefect v1 was that it didn't actually see or move around your code
Yeah, I'm just getting started on Anna G's CI/CD medium posts
The concepts are quite different in v2, but I see how they make it easier to manage deployments now
👍 1
1
Just a lot to absorb
n

Nate

02/23/2023, 7:20 PM
yep! there have certainly been some significant changes let us know if we can help clarify other concepts as you transition also just as an aside, I really like using the
github
flavor of flow storage because then you don't actually need to upload your flow code with
prefect deployment build
(itll just skip upload automatically) since your deployment will just pull from the repo/ref you specify when its time to run - just less to think about in my opinion and it leverages the fact im using git version control anyway
1
j

Jacob Bedard

02/23/2023, 7:23 PM
Is that specifically github? Or possible with gitlab too?
n

Nate

02/23/2023, 7:25 PM
you should be able to find a
gitlab-repository
block in your workspace block catalog that works the same way for gitlab
1
j

Jacob Bedard

02/23/2023, 7:25 PM
Now I'm excited to try it out
Thanks again!
:party-parrot: 1