Ronald Sam
07/11/2023, 5:48 PMChristopher Boyd
07/11/2023, 6:10 PMrun_shell_script
and set_working_dir
deployment.steps to alter the behavior from a git clone, but there isn’t one natively otherwiseRonald Sam
07/11/2023, 6:32 PMChristopher Boyd
07/11/2023, 6:34 PMcat prefect.yaml
# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: path_and_entry_flow
prefect-version: 2.10.18
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/><repo>/Samples.git
branch: main
access_token: null
- prefect.deployments.steps.run_shell_script:
id: test
script: ls -l
stream_output: true
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: communitypath
description: null
flow_name: null
entrypoint: ./Prefect/hello_world.py:hello_world
parameters: {}
work_pool:
name: kubernetes
work_queue_name: null
job_variables:
image: chaboy/test:community
(prefect2) (base) christopherboyd@Christophers-MacBook-Pro ~/prefect_flows/path_and_entry_flow
$ vim prefect.yaml
(prefect2) (base) christopherboyd@Christophers-MacBook-Pro ~/prefect_flows/path_and_entry_flow
$ prefect deploy --all --ci
The `--ci` flag has been deprecated. It will not be available after Dec 2023. Please use the global
`--no-prompt` flag instead: `prefect --no-prompt deploy`.
? Would you like to build a custom Docker image for this deployment? [y/n] (n): n
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Deployment 'hello-world/communitypath' successfully created with id '1c929b72-3b3a-4511-8181-fd9bce8a1263'. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
To execute flow runs from this deployment, start a worker in a separate terminal that pulls work from the
'kubernetes' work pool:
$ prefect worker start --pool 'kubernetes'
To schedule a run for this deployment, use the following command:
$ prefect deployment run 'hello-world/communitypath'
(prefect2) (base) christopherboyd@Christophers-MacBook-Pro ~/prefect_flows/path_and_entry_flow
$ prefect deploy --all --ci
The `--ci` flag has been deprecated. It will not be available after Dec 2023. Please use the global
`--no-prompt` flag instead: `prefect --no-prompt deploy`.
? Would you like to build a custom Docker image for this deployment? [y/n] (n): n
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Deployment 'hello-world/communitypath' successfully created with id '1c929b72-3b3a-4511-8181-fd9bce8a1263'. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
To execute flow runs from this deployment, start a worker in a separate terminal that pulls work from the
'kubernetes' work pool:
$ prefect worker start --pool 'kubernetes'
To schedule a run for this deployment, use the following command:
$ prefect deployment run 'hello-world/communitypath'
Ronald Sam
07/11/2023, 6:39 PM# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: PrefectDemo
prefect-version: 2.10.20
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
repository: https://gitlab.com/.../DevOps.git
branch: PrefectDemo
credentials: "{{ prefect.blocks.gitlab-credentials.rs-git-credentials }}"
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: test_2
version: null
tags: []
description: null
schedule: {}
flow_name: null
entrypoint: .\AppScripts\PrefectDemo\demo.py:my_favorite_function
parameters: {}
work_pool:
name: aws-dev-work-pool
work_queue_name: null
job_variables: {}Christopher Boyd
07/11/2023, 6:42 PMRonald Sam
07/11/2023, 6:44 PMChristopher Boyd
07/11/2023, 6:49 PM./Prefect/hello_world:hello_world
- that is where my flow lives within the repository, but I just have a local folder structure like that, no repoRonald Sam
07/11/2023, 7:04 PMChristopher Boyd
07/11/2023, 7:12 PMprefect deploy
is how you register your flow / deployment
prefect deployment run
(either through CLI, or from UI) triggers a flow run, which is execution, and does a git clone.
Even if I ran this locally, the local agent creates a temporary folder and does a git clone in the temp folder, so I could re-execute 100 times locally and never have it clashRonald Sam
07/11/2023, 7:12 PMChristopher Boyd
07/11/2023, 7:13 PMRonald Sam
07/11/2023, 7:14 PMChristopher Boyd
07/11/2023, 7:14 PMRonald Sam
07/11/2023, 7:14 PMChristopher Boyd
07/11/2023, 7:19 PMRonald Sam
07/11/2023, 7:20 PMChristopher Boyd
07/11/2023, 7:20 PMRonald Sam
07/11/2023, 7:21 PMChristopher Boyd
07/11/2023, 7:22 PMRonald Sam
07/11/2023, 7:22 PMChristopher Boyd
07/11/2023, 7:22 PMRonald Sam
07/11/2023, 7:23 PMChristopher Boyd
07/11/2023, 7:38 PMRonald Sam
07/11/2023, 7:40 PMChristopher Boyd
07/11/2023, 7:40 PMRonald Sam
07/11/2023, 7:41 PMChristopher Boyd
07/11/2023, 7:41 PMRonald Sam
07/11/2023, 7:42 PMChristopher Boyd
07/11/2023, 7:42 PMRonald Sam
07/11/2023, 7:43 PMChristopher Boyd
07/11/2023, 8:00 PMRonald Sam
07/11/2023, 8:00 PMChristopher Boyd
07/12/2023, 6:22 PM- name: hello_world_local
entrypoint: ./Prefect/hello_world.py:hello_world
work_pool:
name: local_test
work_queue_name: null
job_variables: {}
- name: hello_world_k8s
entrypoint: ./Prefect/hello_world.py:hello_world
work_pool:
name: kubernetes
work_queue_name: null
job_variables: {}
Ronald Sam
07/12/2023, 6:24 PMChristopher Boyd
07/12/2023, 6:25 PMRonald Sam
07/12/2023, 6:26 PMChristopher Boyd
07/12/2023, 6:36 PMRonald Sam
07/12/2023, 6:38 PMTrevor Sweeney
09/18/2023, 8:45 PM