and I'm using the a seperate environmetn for deploy and execution

it's strange there isn't a way to do a git pull after a git clone is performed in subsequent deploys @Taylor Curran

unless I'm doing something unusual

for example, I have the following prefect.yaml - I am cloning teh flow from my repo:

cat prefect.yaml
# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: path_and_entry_flow
prefect-version: 2.10.18

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/><repo>/Samples.git
    branch: main
    access_token: null
- prefect.deployments.steps.run_shell_script:
    id: test
    script: ls -l
    stream_output: true

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: communitypath
  description: null
  flow_name: null
  entrypoint: ./Prefect/hello_world.py:hello_world
  parameters: {}
  work_pool:
    name: kubernetes
    work_queue_name: null
    job_variables:
      image: chaboy/test:community

I can deploy it repeatedly:

(prefect2) (base)  christopherboyd@Christophers-MacBook-Pro  ~/prefect_flows/path_and_entry_flow 
 $ vim prefect.yaml
(prefect2) (base)  christopherboyd@Christophers-MacBook-Pro  ~/prefect_flows/path_and_entry_flow 
 $ prefect deploy --all --ci
The `--ci` flag has been deprecated. It will not be available after Dec 2023. Please use the global
`--no-prompt` flag instead: `prefect --no-prompt deploy`.
? Would you like to build a custom Docker image for this deployment? [y/n] (n): n
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Deployment 'hello-world/communitypath' successfully created with id '1c929b72-3b3a-4511-8181-fd9bce8a1263'.  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


To execute flow runs from this deployment, start a worker in a separate terminal that pulls work from the
'kubernetes' work pool:

        $ prefect worker start --pool 'kubernetes'

To schedule a run for this deployment, use the following command:

        $ prefect deployment run 'hello-world/communitypath'

(prefect2) (base)  christopherboyd@Christophers-MacBook-Pro  ~/prefect_flows/path_and_entry_flow 
 $ prefect deploy --all --ci
The `--ci` flag has been deprecated. It will not be available after Dec 2023. Please use the global
`--no-prompt` flag instead: `prefect --no-prompt deploy`.
? Would you like to build a custom Docker image for this deployment? [y/n] (n): n
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Deployment 'hello-world/communitypath' successfully created with id '1c929b72-3b3a-4511-8181-fd9bce8a1263'.  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

To execute flow runs from this deployment, start a worker in a separate terminal that pulls work from the
'kubernetes' work pool:

        $ prefect worker start --pool 'kubernetes'

To schedule a run for this deployment, use the following command:

        $ prefect deployment run 'hello-world/communitypath'

that repo isn’t even cloned or local

the pull step only happens in the container at code execution

I'm actually not using containers but deploying to ec2 instances using gitlab

I run this command to deploy... prefect deploy .\AppScripts\PrefectDemo\demo.py:my_favorite_function

this is my prefect.yaml file

# Welcome to your prefect.yaml file! You can you this file for storing and managing

# configuration for deploying your flows. We recommend committing this file to source # control along with your flow code. # Generic metadata about this project name: PrefectDemo prefect-version: 2.10.20 # build section allows you to manage and build docker images build: null # push section allows you to manage if and how this project is uploaded to remote locations push: null # pull section allows you to provide instructions for cloning this project in remote locations pull: - prefect.deployments.steps.git_clone: repository: https://gitlab.com/.../DevOps.git branch: PrefectDemo credentials: "{{ prefect.blocks.gitlab-credentials.rs-git-credentials }}" # the deployments section allows you to provide configuration for deploying flows deployments: - name: test_2 version: null tags: [] description: null schedule: {} flow_name: null entrypoint: .\AppScripts\PrefectDemo\demo.py:my_favorite_function parameters: {} work_pool: name: aws-dev-work-pool work_queue_name: null job_variables: {}

when I run this commd to run the deployment it performs a git clone every time which fails on the subsequent runs because the git directory already exists in my execution environments working directory.. prefect deployment run 'my-favorite-function/test_2'

i think it was a recent update in the latest prefect version

That pull step is only ran on execution - I don’t even have that repository on my system, I just mimicked the path for the entrypoint

E.g, my entrypoint of

./Prefect/hello_world:hello_world

- that is where my flow lives within the repository, but I just have a local folder structure like that, no repo

there is no git clone happening until it’s executed

in my execution environment

what's your process flow if you update your flow code and push to a feature branch? Won't your execution environment have to perform a git pull to get the most recent changes?

I get this error..

so it only gets cloned there which is why it clashes

I use mac, but when I run a flow run, the agent creates a folder like /var/tmp/123412ioufas0cvs1/prefect

where that string is always random

so 100 flow runs local create 100 random temp folders

thanks

I just don't want so many copies of my repo

I'll also give you the error message I get when I don't preset the working directory

nvm not preseting the working directory works fine now

must've been something else that was causing the issue that I fixed already

raise*

I think that’s a fair idea

mind*

that would be great, thanks Christopher

and for all the help!

that’s not a field for the worker is it?

image.png

wow

Yea, I’ll raise a feature request

@Christopher Boyd, working with it more I think I'm okay with execution doing a git clone in a temp dir, however, I would like to specify where these temp folders are created, my C: drive is really full and would prefer it to be created in the d drive instead but am currently not able to do so because of the issue with git clone not working in more than one run when specifying the working directory

Also, last question, is there a way for me to specify a pull step depending on the work-pool

So you have one pull step for multiple deployments , but not multiple pull steps (within the same prefect.yaml) I don’t think, but I’m not 100% on that particular case

maybe it’s possible, I haven’t tried that

for example, I have 3 env 1. dev 2. qa 3. prod

you would have a separate prefect.yaml because presumably the prefect.yaml is tied to the repo / branch itself

oh gotcha

I’d have to think about it - if you could maybe spell it / detail what that use case looks like for you, or the ideal scenario I can see if it’s possible

Hi @Christopher Boyd, here's the scenario/use case

We have 4 enviornments: dev, stg, qa, prod and we’re using gitlab ci/cd to perform deployments to stg, qa, prod. For dev it’s a playground environment for our developers as a result, they’ll perform their own deployments to the environment. Also, we have python, nodejs, sql, SSIS, batch scripts and etc. used for our data pipelines as a result, we’ll continue to need and use gitlab ci/cd to perform our deployments to stg,qa,prod. For dev I’m thinking to utilize prefect deployment to clone a feature branch to the dev server (execution environment) in each developers computer (dev environment) they’ll be triggering the deployment and execution. For the other environments, the deployment will be done by gitlab as a result, we only need prefect to execute. The deployment step using prefect will just be to use the local pull step to tell prefect where the script is located when the execution is triggered. Each of the environments will have a worker pool installed, since they’re each an execution environment. To perform the above process, I think I’ll need a pull step for each worker-pool to specify what I want it to do when the flow is executed in a worker-pool. For example, in dev, I’d like it to perform a git clone to a temp folder at a specific, working directory. For stg, qa, prod, I’d like it to look for the flow at this local folder path location in each of the environment. Is this setup currently possible?

Hi @Christopher Boyd, by any chance did you have time to review the above use case? I don't think we can go live with our flows until it addresses worker-pool specific pull steps.

but regarding different pull steps, no - you could use a run_shell_script to do git pull behavior if that’s intended, but you can’t tie one pull step to one deployment, and another to another deployment

I think this is a good example repo:

for dev we want to perform a git clone pull step but for all the other environments, since gitlab is already helping us with the deployment to the environment, we just want to use the prefect local pull step to specify where the file is located in the execution environment

I do think maybe there is some opportunity to use a flag for a git pull in lieu of git clone, but that comes with its own challenges (like it would only be for local process types probably) because it wouldn’t make sense on k8s

this is a different issue where we'd like to specify the deployment based on the execution environment

because depending on the environment deployment is already handled by gitlab or not

for dev deployment isn't handled by gitlab, so we want to have git clone as the pull step, while for the other environments, we just want to put prefect to where the flows are located in the execution environment

in that case, it’s just a local file path

do dev needs to do a git clone, but for prod, if gitlab handles the code existing there and you don’t need to git clone, then your pull step can just be null, and your path / entrypoint are all you need to specify

but doesn't all environment share the same prefect.yaml file?