Is there a simple end to end example of deploying ...
# prefect-cloud
b
Is there a simple end to end example of deploying a flow on cloud? I have an example running locally (after starting the server locally). In my case, I want the simple flow to be scheduled once a day. It's only requirement is
requests
I can appreciate all of the knobs that can be controlled, but I just want to see something running on a schedule before I tune it or configure it.
FWIW, I followed the instructions on the Cloud dashboard. Effectively something like auth, create a .py script and its flow.serve. The issue here is that the script is running. I see it on Cloud, but if I ctrl+C locally, things are impacted. Trying to build intuition because clearly this can't be a way teams push from local to Cloud.
I got something working, and things make a lot more sense to me now, so logging here in case it helps someone else. 1. After creating an example flow, I used
prefect deploy
which opens a pretty slick CLI tool. 2. I followed the options and selected the basics a. My defined flow. It even found this in a nested
/flows
folder b. Left the name default c. configure schedules -> i. Kinda odd because I defined this in the script (below). I see what the CLI is doing now, but basically I am still not sure why I would ever set it in the py file. ii. selected CRON and moved on d. activated the schedule when prompted e. created a worker pool and gave a name f. infra type -> prefect managed g. Remote storage, no as this is found on my local machine h. save config -> yes The last step above is where my šŸ’” went off. The YAML file shows why those options matter and now that I see the resource on Prefect Cloud, dots are coming together around ways to store your flows or the level of control you have over the worker pool.
Copy code
from prefect import flow


@flow(log_prints=True)
def hello_world(name: str = "world", goodbye: bool = False):
    print(f"Hello {name} from Prefect! šŸ¤—")

    if goodbye:
        print(f"Goodbye {name}!")


if __name__ == "__main__":
    hello_world.deploy(
        name="my-first-deployment",
        cron="15 0 * * *"
    )
In the end, some things were not obvious (to me) and only really started to make sense relative to the docs after stumbling across the CLI flow, which is fantastic. hattip
šŸ‘ 1
However, I can't seem to get a trigger of the flow to run successfully. I am getting crashed errors in the log. Kinda stumped. When I run
python myfile.py
it logs successfully in cloud. Need to put this aside, but I am missing something about the free tier and worker pools.
b
Hi Brock, really appreciate the organic stream of consciousness you added here. Outlines like this really help us better understand friction points in the onboarding process.
1. Kinda odd because I defined this in the script (below). I see what the CLI is doing now, but basically I am still not sure why I would ever set it in the py file.
In short, you came across two ways of scheduling a deployed flow (through the CLI using
prefect deploy
, and
.deploy()
which uses the python SDK). You really only had to do one or the other, as they can both be used to: 1) create a scheduled deployment and 2) assign work to a work pool.
If you wouldn't mind sharing your error from the logs, we'll help you get to the next step.
n
hey @Brock - just to add on, i made a

videoā–¾

essentially exactly about this tension > create a .py script and its flow.serve. The issue here is that the script is running. I see it on Cloud, but if I ctrl+C locally, things are impacted. Trying to build intuition because clearly this can't be a way teams push from local to Cloud. which may be useful if you're into videos, as it goes into some detail on the suggestions bianca mentioned!
šŸ”„ 1
b
Many thanks to you both! Yes, I wanted to log my learnings in case it helped, because the docs are thorough, but I found myself jumping around in order to identify what was happening or why certain things mattered. In short, it does seem like the prefect deploy with a YAML and .deploy were tripping me up out of the gate. Having said that, this morning I was looking through the docs again and perhaps one bit I was completely ignoring is that for the free tier on Prefect cloud, it appears that you require that my code live somewhere it can be referenced. The mental model I started with was that
prefect deploy
would push my code to Cloud and that it would live there, with additional deployments changing/updating the code and configuration as needed. That doesn't appear to be the case. I am referencing this link here: https://docs.prefect.io/3.0/deploy/infrastructure-examples/managed#code-storage. I haven't gone down this path yet. @Nate I am giving the video a watch now. Because this is for my course, I am trying to keep the deployment process as simple as possible and minimize the moving parts they have to worry about. Thanks again, I really appreciate it.
And to be more specific, I am getting "Crashed" with
Copy code
Reported flow run '20480205-73a3-458f-b5b7-e7d4b072b10e' as crashed: Flow run process exited with non-zero status code 1.
As noted above, I now wonder if its because I am pointing to a local source and not, say a public github repo. In practice I wouldn't want anything public, but for the course, this is fine.
To come back to this, I just created a new deployment with a very simple flow found on a public git repo and it is still crashing. I figured my issue was that I wasn't pointing to a public source to pull from. Completely stumped @Nate @Bianca Hoch This is the function: https://github.com/Btibert3/BA882-Fall24-InClass-Project/blob/main/prefect/sandbox/hello-world.py My yaml for deployment
Copy code
# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: prefect
prefect-version: 3.0.4

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/Btibert3/BA882-Fall24-InClass-Project.git>
    branch: main

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: hello-world
  version: null
  tags: []
  concurrency_limit: null
  description: null
  entrypoint: sandbox/hello-world.py:simple_flow
  parameters: {}
  work_pool:
    name: brock-worker1
    work_queue_name: null
    job_variables: {}
  enforce_parameter_schema: true
  schedules:
  - cron: 0 0 * * *
    timezone: UTC
    day_or: true
    active: true
    max_active_runs: null
    catchup: false
Clearly I am missing something obvious or silly, but now that I am using pull from a public project, I have zero idea why its crashing.
n
the
entrypoint
should always be relative to the root of the storage, which in this case is the repo so I think in your yaml you'd want
Copy code
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: hello-world
  ...
  entrypoint: prefect/sandbox/hello-world.py:simple_flow
if you look at your worker logs, I imagine you'd see some
FileNotFoundError
b
Thanks for the quick response Nate! Is there some other place to look for the worker logs? It's not obvious to me in the UI. In the screenshot above, that is all I have to go on, unless I am overlooking something.
n
Is there some other place to look for the worker logs?
yes! itll be where you ran
prefect worker start
b
That would assume I started a worker. šŸ™‚
n
indeed! are you not using a worker?
b
My process was to write the flow, and then use
prefect deploy
and use the guided CLI, which created a worker pool for me b/c last evening after class I didn't have one setup. I just left it at that
n
aha. if you're using managed infra then we run the worker and you would not need to. another consequence of that is you can't dig through the logs of the worker (bc its in our infra) it sounds like we should do a better job surfacing errors from managed workers, but as far as your current situation goes: • i still suspect the
FileNotFoundError
bc entrypoints must be relative to repo root and it appears yours was not • if you want to dig around yourself, you could switch your
work_pool
name
in your yaml to
local
after running
prefect worker start -p local
which would just create you a process work pool and have the worker start listening for runs sent to that pool
b
cool, I will play around. The idea is to use the managed infra for the class to remove how many things the students would need to think about right now.
šŸ‘ 1
n
fwiw, my personal opinion is that
.serve
is by far the easiest way to get yourself a scheduled deployment that has the deployment mechanics - because it doesnt force you to engage with workers, work pools or any of that at all. you just get your flow function working, and then
my_flow.serve()
at the bottom of the file and
python my_file.py
work pools and workers are only necessary when you need: • dynamic dispatch of infra (think kubernetes, ECS, docker) • to decouple code authoring from infra config (e.g. data science team + dev ops team) you can daemonize
.serve
if you dont want the process to die when you close your computer
upvote 1
b
Sweet, I got this running. You nailed it above with the path being incorrect relative to the repo. I was trying to be "smart" and started the CLI flow inside the prefect folder, which is why it was off. I started over: • prefect deploy at the root of the project • chose the remote worker pool - managed infra • got that yaml file • and gave it a go via
prefect deployment run 'simple-flow/hello-world'
I am off to the races, I believe. I just need to figure out how to include pip packages and this should be enough for the class to get off the ground. Thank you for your help and incredible patience, I appreciate it.
catjam 2
n
sure thing - glad you got things working!
hattip 1