AlexPerez
08/06/2023, 2:27 PMcloud-run
work pool that points to the same Artifact as my GCP Cloud Run Job
Block
, however my queue will remain unhealthy
until I spin that local agent. This is obviously preventing me from having a more ‘production’ solution.
Does anyone have experience with this setup and could help me out? My Dockerfile
is in the 🧵FROM prefecthq/prefect:2.6.5-python3.10
# Set the working directory inside the container
WORKDIR /app
# Copy the local Python files into the container's working directory
COPY ./prefect/tasks/prefect_tasks.py /app/tasks/prefect_tasks.py
RUN ["pip", "install", "gcsfs", "numpy", "pandas", "scrapy", "pandas-gbq", "prefect-gcp[cloud_storage]", "unidecode"]
CMD ["prefect", "agent", "local", "start"]
morten
08/06/2023, 3:40 PMAlexPerez
08/06/2023, 3:46 PMCloud Run:push
. As soon as I created this type of pool, my flow works like flawlessly.
Hi Morten 🙂 I wanted to start an agent, I just did not notice that when working with Serverless Computing Infrastructure, I cannot create a pull of type Google Cloud Run
. I needed instead the Cloud Run:push
Thanks for the quick answer, I am still struggling with the therminology I only started with Prefect this Friday, lots to digest. Can you point to some good resource to understand agent Vs worker?morten
08/06/2023, 3:54 PMrequirements.txt
install step:
pull:
- prefect.deployments.steps.git_clone:
id: clone-step
repository: <https://github.com/org/repo.git>
- prefect.deployments.steps.pip_install_requirements:
directory: {{ clone-step.directory }}
requirements_file: requirements.txt
stream_output: False
from https://docs.prefect.io/2.11.3/concepts/deployments/#utility-stepsclone
is a bit weird, as it clones into an empty dir, but if you already ran the code once and the repo dir already exists, it will throw and error.Nate
08/06/2023, 7:00 PMmy queue will remainit makes sense that your work queue would be unhealthy until you start the worker, as a healthy work queue is one that can successfully poll a worker (or agent). typically, you'll want to start a worker on an VM with something like systemd so that its always running. note that anyuntil I spin that local agent. This is obviously preventing me from having a more ‘production’ solution.unhealthy
Push
type work pool will allow you to execute flow runs from a deployment without running a worker at all, since with the push type work pool, Prefect cloud will spin up the worker for you when its time for a deployment's flow run to be executed - if you want to use the normal Cloud Run work pool, you'll just have to run a worker as i mentioned such that it can submit jobs to your cloud run environment
as for the difference between workers and agents, the tldr is that workers are strongly-typed, next gen agents. "strongly-typed" meaning that only X typed workers can pull work from X typed work pools (e.g. Cloud Run worker pulls from Cloud Run work pool). With workers, you don't need to attach an infra block (like GCP Cloud Run) to your deployments, as workers will grab the job infra template the flow run needs from the work pool instead
workers (of any type) can also execute pull
steps, as @morten illustrates above, which is typically when your worker fetches the flow code (but can also do any arbitrary setup just before flow run execution, like pip_install_requirements
if you need to)