@Marvin If I’m developing and want to test a deployment with my newly updated code, how do I go about deploying (using

prefect deploy --all

) and have it include the updated code. right now I’m getting this during the docker image build

Step 5/6 : WORKDIR /opt/prefect/flows/
 ---> Using cache
 ---> 07afdee1b173

for context, I’m using the kubernetes worker pool on gke. What are the best practices here?

To deploy your updated code using Prefect, you can use the

prefect deploy --all

command. This command will deploy all deployments. If you want to deploy specific deployments, you can use the

--name

flag followed by the name of your flow and deployment (source). Regarding the Docker image build, once you have a Dockerfile, you can build your image by running

docker build -t your-image-name .

. Make sure your Dockerfile includes the necessary commands to copy your updated code into the image. You can check that your build worked by running a container from your new image. The container will need an API URL and an API key to communicate with Prefect Cloud. You can get an API key from the API Keys section of the user settings in the Prefect UI (source). For your Kubernetes worker pool on GKE, you will create a Docker type work pool via the CLI. Using the Docker work pool type means that all work sent to this work pool will run within a dedicated Docker container using a Docker client available to the worker (source). Please note that these are general guidelines and you may need to adjust them based on your specific setup and requirements.

For your Kubernetes worker pool on GKE, you will create a Docker type work pool via the CLI

🙅 there's the k8s work pool for that, but I suspect you already know that

I believe I’m baking it in, I haven’t set up anything for git. Though that sounds like it may be a better method

so historically I'd say "best practice" depend on your org / constraints, but personally what I prefer to manage (when using k8s runtime) is: • write some

Dockerfile

that does

pip install .

or

pip install -r requirements.txt

or whatever your flow would need for deps • either use a `build` step to build this image, or build / push the image in your own CI / image pipeline if preferred. either way, set that

image

as a

job_variable

on your work pool directly, or override the `image` on the work pool _for this specific deployment_ • use a git clone pull step to pull your flow source code down from your repo at runtime on top of your

image

built in above steps the thing i like about this is that, most often for me, my flow source code changes much more often than

image

/ deps need to, i.e. i shouldn't have to rebuild my

image

just bc i add a comment in my flow, for example. with the above, all i have to do is push my code to github / gitlab etc like I normally would anyways, and then thats the code thats gonna get pulled down at runtime only gotcha there is that you actually have to remember to push your code 🙂 there's been times where I make local changes to my flow and test and accidentally use the version of the flow in

main

bc may changes hadnt been pushed, but for prod deployments that shouldnt be such a gotcha

Ok, that definitely makes sense. So one quick follow up. If I go the “build step” route, what exactly would cause it to rebuild, and what things wouldn’t cause to to rebuild, e.g. using the cache instead?

if you have build steps on all of them, and you run them independently

prefect --no-prompt deploy -n some-deploy

, then docker will use cached layers the same way as it would if you did

docker build

n times in a row im pretty sure if you say

prefect deploy --all

and all of them use the same

build

step, then we'll only run

docker build

on your behalf one time, but I'm not 100% on that off the top

Ok cool, sounds good. Thanks! I’ll give that a try and see how it goes. I’m just converting our first pipeline over, so I’m still learning a lot

cool! np - feel free to ask anything else if you hit some blockers

What changes do I have to make so that the docker build portion skips the

COPY

section, since I have the pull part now?

you can specify your own `Dockerfile` if you want, where you could just not include any

COPY

of your source code

ok cool, that makes sense

did you call .save on that block? you can do prefect block ls to see what blocks you’ve created on your current server

I did it in the cloud UI, looks like they are there, though I’m not using the repo one in the yaml file

ahhh you need to out the type before the block document name, so that slug on the right but dot delimited instead of /

ahhh, ok I'll give that a try

hmm can you show what you tried?

Yeah, here’s my pull section

pull:
- prefect.deployments.steps.run_shell_script:
    id: get-git-branch
    script: git rev-parse --abbrev-ref HEAD
    stream_output: false
- prefect.deployments.steps.git_clone: &clone_repo
    repository: <https://gitlab.com/org/repo.git>
    branch: "{{ get-git-branch.stdout }}"
    access_token: "{{ prefect.blocks.secret.gitlab-credentials.gitlab-access-token }}"

ahh so you'll want

prefect.blocks.gitlab-credentials.gitlab-access-token

because its

prefect.blocks.{block-type}.{block-document-name}

where

secret

is just a common type of block, often templated in pull steps like this. your block is type

gitlab-credentials

ohhh. man, I'm struggling, haha. thanks

no problem! feel free to comment on what you think should be more prominent in the docs, there are certain aspects of this that we'd like to make more obvious 🙂

cool, yeah I could think about some feedback when I get it. unfortunately, haha, its still erroring out

File "/usr/local/lib/python3.10/site-packages/prefect/runner/storage.py", line 615, in _format_token_from_credentials
    if not user_provided_token.startswith("oauth2:")
AttributeError: 'dict' object has no attribute 'startswith'

I just pasted in the access token string under "Token" in the block. I'm looking here: https://docs.prefect.io/latest/guides/deployment/storage-guide/#__tabbed_1_3 And I'm not sure what "In addition to the block name, most users will need to fill in the GitLab Username and GitLab Personal Access Token fields." refers to, in step 4. And its also not clear to me what the

prefect block register

command does, vs. adding them in the cloud UI

oh wait, i actually think this could be a bug we just fixed today related specifically to gitlab / github creds blocks here as a sanity check, if you

pip install git+<https://github.com/PrefectHQ/prefect.git>

does it work?

oh ok, I'll install that and redeploy and run

oh yeah sorry, i forgot you're on gke. yeah if you apt install git or whatever in your dockerfile then you should be able to just put

git+<https://github.com/PrefectHQ/prefect.git>

in your reqs file (until we go ahead and release 2.14.9 with the fix tomorrow) - or you could test it locally if you have like docker desktop / minikube if you want

ahh, yeah i forgot I saw that extra key. I'm trying an install through the req file for now. I'll let you know if it gets past that issue once it runs

ah my bad, didnt think of it right away

no, totally fine, those are great to know for the future

hmm 🧐 - same trace as above?

Yeah, same trace. And it did get the new req

Collecting prefect@ git+<https://github.com/PrefectHQ/prefect.git>

during the build

2.14.8+4.g3c8f95331

yeah this "dirty" version is expected if you install from main like that mm there's likely some issue with the credentials value (or its type) provided if it doesnt work on main all you're doing is templating in a gitlab creds block into your pull step right?

pull:
- prefect.deployments.steps.run_shell_script:
    id: get-git-branch
    script: git rev-parse --abbrev-ref HEAD
    stream_output: false
- prefect.deployments.steps.git_clone: &clone_repo
    repository: <https://gitlab.com/org/repo.git>
    branch: "{{ get-git-branch.stdout }}"
    access_token: "{{ prefect.blocks.gitlab-credentials.gitlab-access-token }}"

i can try to reproduce this later if this is all it takes to produce the error?

Yes, exactly that. I did just try to reproduce the code locally a bit, and it seems fine locally..

from prefect_gitlab import GitLabCredentials

gitlab_credentials_block = GitLabCredentials.load("gitlab-access-token")
credentials = gitlab_credentials_block.dict()
credentials["token"] = credentials["token"].get_secret_value()
print(credentials["token"]

Then it prints the token value as a string

if you have the chance, id be curious to see if everything works if you (install main and) change your deployments work pool to point at some local process pool. since the cloning / creds part should exactly the same

Yeah I can try that. I don’t have a local worker setup yet, but I can try it out

i mean, i was thinking you could just do a process pool and just say

prefect worker start

from your dev venv

Oh, nice, I don’t think I was aware of that, I can give it a try, let me check the docs

hmmmm interesting. where did you print that from?

It looked like that here: https://github.com/PrefectHQ/prefect/blob/3c8f953316435f7dfd44a7c21a7ae1ea531c66e6/src/prefect/runner/storage.py#L165C1-L165C1 and here: https://github.com/PrefectHQ/prefect/blob/3c8f953316435f7dfd44a7c21a7ae1ea531c66e6/src/prefect/runner/storage.py#L581

and you have main installed?

As far as I can tell

Name: prefect
Version: 2.14.8+4.g3c8f95331
Summary: Workflow orchestration and management.
Home-page: <https://www.prefect.io>
Author: Prefect Technologies, Inc.
Author-email: <mailto:help@prefect.io|help@prefect.io>

I feel like I just have something off though..

try this instead

pull:
- prefect.deployments.steps.run_shell_script:
    id: get-git-branch
    script: git rev-parse --abbrev-ref HEAD
    stream_output: false
- prefect.deployments.steps.git_clone: &clone_repo
    repository: <https://gitlab.com/org/repo.git>
    branch: "{{ get-git-branch.stdout }}"
    credentials: "{{ prefect.blocks.gitlab-credentials.gitlab-access-token }}"

ahhh, k trying that now

haha its all good! i forgot we accept both the bare

access_token

as well as

credentials

for blocks there what exactly is your intention, like what branch do you want to be cloning / how do you want to decide that?

does that branch logic I have seem right?

I’m wanting it to clone the branch that I’m currently developing on, so that I can have it pull that branch and reflect my current development updates

oh yeah i think the git rev-parse command lgtm, i would just expect that bc

stream_output

is false, that

get-git-branch.stdout

is empty and we default to

main

oh ok, I was following the logic for adding the git hash to the image tag, which had stream_output as false

oh yeah im wrong, it works for me with

stream_output: false

if i switch to some

test

branch

- name: host-env-test
  entrypoint: src/demo_project/read_env.py:read_env
  build:
  - prefect.deployments.steps.run_shell_script:
      id: get-branch
      script: git rev-parse --abbrev-ref HEAD
      stream_output: false
  - prefect.deployments.steps.run_shell_script:
      script: echo "we're on branch {{ get-branch.stdout }}"

gives

(prefect-monorepo) pad-2 :: src/demos/prefect-monorepo ‹test›
» prefect deploy -n host-env-test

Running deployment build steps...
 > Running run_shell_script step...
 > Running run_shell_script step...
we're on branch test

ahhhh, ok I moved it to my build step and it seems to be working now. I didn’t realize you could pull the output from build inside the pull part