https://prefect.io logo
Channels
Powered by
a

AlexPerez

08/06/2023, 2:27 PM
Hi all, I am bit stuck and I would love some community help 🙏 My current prefect personal setup is using prefect cloud and a docker image hosted as an Artifact in GCP (I mainly followed this article). I can run my deployments as long us I spin a local agent. I already created a 
cloud-run
work pool that points to the same Artifact as my 
GCP Cloud Run Job
Block
, however my queue will remain 
unhealthy
until I spin that local agent. This is obviously preventing me from having a more ‘production’ solution. Does anyone have experience with this setup and could help me out? My 
Dockerfile
is in the 🧵
✅ 1
Copy code
FROM prefecthq/prefect:2.6.5-python3.10
# Set the working directory inside the container
WORKDIR /app
# Copy the local Python files into the container's working directory
COPY ./prefect/tasks/prefect_tasks.py /app/tasks/prefect_tasks.py
RUN ["pip", "install", "gcsfs", "numpy", "pandas", "scrapy", "pandas-gbq", "prefect-gcp[cloud_storage]", "unidecode"]

CMD ["prefect", "agent", "local", "start"]
m

morten

08/06/2023, 3:40 PM
are you starting an agent or worker?
and "hi"- of course 🙂
a

AlexPerez

08/06/2023, 3:46 PM
So, the main problem I had is that my work pool was not of type 
Cloud Run:push
. As soon as I created this type of pool, my flow works like flawlessly. Hi Morten 🙂 I wanted to start an agent, I just did not notice that when working with Serverless Computing Infrastructure, I cannot create a pull of type 
Google Cloud Run
. I needed instead the 
Cloud Run:push
Thanks for the quick answer, I am still struggling with the therminology I only started with Prefect this Friday, lots to digest. Can you point to some good resource to understand agent Vs worker?
m

morten

08/06/2023, 3:54 PM
I just started a couple of weeks ago. I dont have a good answer for what the difference is between agent and worker. I know that they have used the term "agent" for years, but I think "worker" is relatively new.
It also seems like their Serverless Computing Infrastructure lacks some documentation - I am getting a blank section, where I think there should be something
one thing that might be nice to do, is using their 
requirements.txt
install step:
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        id: clone-step
        repository: <https://github.com/org/repo.git>
    - prefect.deployments.steps.pip_install_requirements:
        directory: {{ clone-step.directory }}
        requirements_file: requirements.txt
        stream_output: False
from https://docs.prefect.io/2.11.3/concepts/deployments/#utility-steps
in case you dont want to run the pip install in the docker
note: the 
clone
is a bit weird, as it clones into an empty dir, but if you already ran the code once and the repo dir already exists, it will throw and error.
gratitude gracias 1
n

Nate

08/06/2023, 7:00 PM
hi @AlexPerez
my queue will remain 
unhealthy
until I spin that local agent. This is obviously preventing me from having a more ‘production’ solution.
it makes sense that your work queue would be unhealthy until you start the worker, as a healthy work queue is one that can successfully poll a worker (or agent). typically, you'll want to start a worker on an VM with something like systemd so that its always running. note that any 
Push
type work pool will allow you to execute flow runs from a deployment without running a worker at all, since with the push type work pool, Prefect cloud will spin up the worker for you when its time for a deployment's flow run to be executed - if you want to use the normal Cloud Run work pool, you'll just have to run a worker as i mentioned such that it can submit jobs to your cloud run environment as for the difference between workers and agents, the tldr is that workers are strongly-typed, next gen agents. "strongly-typed" meaning that only X typed workers can pull work from X typed work pools (e.g. Cloud Run worker pulls from Cloud Run work pool). With workers, you don't need to attach an infra block (like GCP Cloud Run) to your deployments, as workers will grab the job infra template the flow run needs from the work pool instead workers (of any type) can also execute 
pull
steps, as @morten illustrates above, which is typically when your worker fetches the flow code (but can also do any arbitrary setup just before flow run execution, like 
pip_install_requirements
if you need to)
gratitude gracias 1
Powered by