Brock
10/03/2024, 10:15 PMrequests
I can appreciate all of the knobs that can be controlled, but I just want to see something running on a schedule before I tune it or configure it.Brock
10/04/2024, 3:02 AMBrock
10/04/2024, 3:50 AMprefect deploy
which opens a pretty slick CLI tool.
2. I followed the options and selected the basics
a. My defined flow. It even found this in a nested /flows
folder
b. Left the name default
c. configure schedules ->
i. Kinda odd because I defined this in the script (below). I see what the CLI is doing now, but basically I am still not sure why I would ever set it in the py file.
ii. selected CRON and moved on
d. activated the schedule when prompted
e. created a worker pool and gave a name
f. infra type -> prefect managed
g. Remote storage, no as this is found on my local machine
h. save config -> yes
The last step above is where my š” went off. The YAML file shows why those options matter and now that I see the resource on Prefect Cloud, dots are coming together around ways to store your flows or the level of control you have over the worker pool.
from prefect import flow
@flow(log_prints=True)
def hello_world(name: str = "world", goodbye: bool = False):
print(f"Hello {name} from Prefect! š¤")
if goodbye:
print(f"Goodbye {name}!")
if __name__ == "__main__":
hello_world.deploy(
name="my-first-deployment",
cron="15 0 * * *"
)
In the end, some things were not obvious (to me) and only really started to make sense relative to the docs after stumbling across the CLI flow, which is fantastic. hattipBrock
10/04/2024, 4:35 AMpython myfile.py
it logs successfully in cloud.
Need to put this aside, but I am missing something about the free tier and worker pools.Bianca Hoch
10/04/2024, 2:00 PMBianca Hoch
10/04/2024, 2:04 PM1. Kinda odd because I defined this in the script (below). I see what the CLI is doing now, but basically I am still not sure why I would ever set it in the py file.In short, you came across two ways of scheduling a deployed flow (through the CLI using
prefect deploy
, and .deploy()
which uses the python SDK). You really only had to do one or the other, as they can both be used to: 1) create a scheduled deployment and 2) assign work to a work pool.Bianca Hoch
10/04/2024, 2:05 PMNate
10/04/2024, 3:10 PMBrock
10/04/2024, 3:37 PMprefect deploy
would push my code to Cloud and that it would live there, with additional deployments changing/updating the code and configuration as needed. That doesn't appear to be the case. I am referencing this link here: https://docs.prefect.io/3.0/deploy/infrastructure-examples/managed#code-storage. I haven't gone down this path yet.
@Nate I am giving the video a watch now. Because this is for my course, I am trying to keep the deployment process as simple as possible and minimize the moving parts they have to worry about.
Thanks again, I really appreciate it.Brock
10/04/2024, 3:39 PMReported flow run '20480205-73a3-458f-b5b7-e7d4b072b10e' as crashed: Flow run process exited with non-zero status code 1.
As noted above, I now wonder if its because I am pointing to a local source and not, say a public github repo. In practice I wouldn't want anything public, but for the course, this is fine.Brock
10/04/2024, 6:10 PM# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: prefect
prefect-version: 3.0.4
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/Btibert3/BA882-Fall24-InClass-Project.git>
branch: main
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: hello-world
version: null
tags: []
concurrency_limit: null
description: null
entrypoint: sandbox/hello-world.py:simple_flow
parameters: {}
work_pool:
name: brock-worker1
work_queue_name: null
job_variables: {}
enforce_parameter_schema: true
schedules:
- cron: 0 0 * * *
timezone: UTC
day_or: true
active: true
max_active_runs: null
catchup: false
Brock
10/04/2024, 6:11 PMNate
10/04/2024, 6:14 PMentrypoint
should always be relative to the root of the storage, which in this case is the repo
so I think in your yaml you'd want
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: hello-world
...
entrypoint: prefect/sandbox/hello-world.py:simple_flow
Nate
10/04/2024, 6:14 PMFileNotFoundError
Brock
10/04/2024, 6:18 PMNate
10/04/2024, 6:19 PMIs there some other place to look for the worker logs?yes! itll be where you ran
prefect worker start
Brock
10/04/2024, 6:20 PMNate
10/04/2024, 6:21 PMBrock
10/04/2024, 6:21 PMprefect deploy
and use the guided CLI, which created a worker pool for me b/c last evening after class I didn't have one setup. I just left it at thatBrock
10/04/2024, 6:22 PMNate
10/04/2024, 6:25 PMFileNotFoundError
bc entrypoints must be relative to repo root and it appears yours was not
⢠if you want to dig around yourself, you could switch your work_pool
name
in your yaml to local
after running prefect worker start -p local
which would just create you a process work pool and have the worker start listening for runs sent to that poolBrock
10/04/2024, 6:27 PMNate
10/04/2024, 6:29 PM.serve
is by far the easiest way to get yourself a scheduled deployment that has the deployment mechanics - because it doesnt force you to engage with workers, work pools or any of that at all. you just get your flow function working, and then my_flow.serve()
at the bottom of the file and python my_file.py
work pools and workers are only necessary when you need:
⢠dynamic dispatch of infra (think kubernetes, ECS, docker)
⢠to decouple code authoring from infra config (e.g. data science team + dev ops team)
you can daemonize .serve
if you dont want the process to die when you close your computerBrock
10/04/2024, 6:41 PMprefect deployment run 'simple-flow/hello-world'
I am off to the races, I believe. I just need to figure out how to include pip packages and this should be enough for the class to get off the ground.
Thank you for your help and incredible patience, I appreciate it.Nate
10/04/2024, 6:42 PM