https://prefect.io logo
Title
m

Moises Vera

08/13/2022, 6:10 PM
noob question for prefect 2, • is it necessary to run the build command each time I change my code? • looks like the build command changes my deployment file with my old config so I have to re write the old config I had again each time
1
o

Oscar Björhn

08/13/2022, 6:35 PM
1. I believe so! Technically not always, I suppose you could get away with manually uploading your changed .py files to your remote file system, depending on the specifics of your infrastructure. But generally, yes. 2. I believe the idea is that deployment files are temporary. You run your build command, the file is generated, and then you apply the file.
🙌 1
a

Anna Geller

08/13/2022, 6:36 PM
#1 it depends - your deployment only points to the storage location -- if you're using local storage = default, then Prefect will always run the latest flow; if you're using S3 storage though you then either need to rerun build or reupload the changed files yourself -- this is sth what is usually handled as part of CI/CD in a typical development workflow #2 you can ignore YAML if you define everything in your CLI build command
o

Oscar Björhn

08/13/2022, 6:36 PM
Dang, bad timing. Sorry about that.
a

Anna Geller

08/13/2022, 6:37 PM
it's perfect timing and perfect answer
👍 1
m

Moises Vera

08/13/2022, 6:38 PM
thank you guys
do you have a reference of a CI/CD config? so I read it as reference
o

Oscar Björhn

08/13/2022, 6:44 PM
I don't have a full CI/CD config yet but I do have a python helper script I run that does everything for me in one go. I could send a few snippets from that if you think it's of any use?
m

Moises Vera

08/13/2022, 7:27 PM
oh thank you
I appreciate
I just want to know best practices about CI with prefect
o

Oscar Björhn

08/13/2022, 7:38 PM
I wouldn't call what I'm doing best practice, necessarily, but it seems to work alright! It also seems like the Prefect team is working on both simplifying and adding more options to deployments at the moment, so things might change. This is what I have at the moment:
# 3 steps:
# 1. Use docker's python package to build and upload a base image for my flow, if an up-to-date one doesn't already exist. This step has been omitted since it's kind of long and specific to our setup.
# 2. Prepare and run the deployment build command using shell.
# 3. Prepare and run the deployment apply command using shell.

prefect_build_command = f"prefect deployment build orchestration/flows/{flow_name_clean}.py:default -n \"{flow_name} ({env.value})\" -t {env.value} -t {extra_tag} --storage-block azure/azure-{env.value} --infra docker-container --override image={full_image_path} -o {flow_name_clean}-deployment.yaml"
print(prefect_build_command)

with Popen([prefect_build_command], stdout=PIPE, shell=True) as proc:
    print(proc.stdout.read())

prefect_apply_command = f"prefect deployment apply {flow_name_clean}-deployment.yaml"

with Popen([prefect_apply_command], stdout=PIPE, shell=True) as proc:
    print(proc.stdout.read())
:thank-you: 2
It's pretty simple, but maybe it'll give you some inspiration regarding how you can use variables to handle your deployments in a generic manner.
j

James Brady

08/14/2022, 1:26 PM
@Moises Vera re: your "build command changes my deployment file" point above, I was in the same situation but was able to switch to infrastructure blocks (and the
-ib
option) rather than infrastructure (and the
-i
) option and then I didn't ned to edit the YAML after having built a deployment. This is basically the opposite of the solution Anna mentions above, where you specify everything as command line switches. The end result is the same though, of not needing to edit the YAML.
a

Anna Geller

08/14/2022, 2:40 PM
This README clarifies things a bit https://github.com/anna-geller/prefect-deployments