https://prefect.io logo
j

Javier Ochoa

08/30/2023, 6:07 PM
Hello, I have a problem, I remember in the course, the team told us to use workers instead of agents. When I changed my command to use worker I get the following error, So I wonder if I am doing something wrong here:
Copy code
Traceback (most recent call last):
  File "/home/javier/git/nzero/usg-prefect-2/.venv/lib/python3.9/site-packages/prefect/workers/base.py", line 784, in _submit_run
    await self._check_flow_run(flow_run)
  File "/home/javier/git/nzero/usg-prefect-2/.venv/lib/python3.9/site-packages/prefect/workers/base.py", line 770, in _check_flow_run
    raise ValueError(
ValueError: Flow run UUID('53a4f0fc-1802-4986-8a72-a15cbd1f4674') was created from deployment 'my-worker' which is configured with a storage block. Workers currently only support local storage. Please use an agent to execute this flow run.
n

Nate

08/30/2023, 6:09 PM
hey @Javier Ochoa - have you recreated your deployment via
prefect deploy
since switching to
prefect worker start
? the deployment you create with
prefect deploy
will be a little bit different from the one you get with
prefect deployment build
j

Javier Ochoa

08/30/2023, 6:13 PM
I manage our deployments from python code:
Copy code
storage = GitHub.load(settings.DOMAIN_ENV_HOST_ID)
    work_pool_name = (
        f"{settings.DOMAIN}-{settings.ENVIRONMENT}-agent-{settings.LOCALHOST_ID}"
    )
    Deployment.build_from_flow(
        flow=flow_function,
        name=settings.DOMAIN_ENV_HOST_ID,
        tags=[settings.LOCALHOST_ID],
        storage=storage,
        work_pool_name=work_pool_name,
        ignore_file=f"{settings.PROJ_ROOT_PATH}/.prefectignore",
        apply=True,
    )
n

Nate

08/30/2023, 6:22 PM
understood. the block-based deployment pattern (
prefect deployment build
&
Deployment.build_from_flow()
) was designed for agents. workers are designed to pick up and submit deployments created via `prefect.yaml` and `prefect deploy` . though we plan to have a pythonic interface for worker / workpool deployments soon, we dont have one yet. so if you'd like to define your deployments in python for now, I might just stay on agents until we have a python interface released that's designed with workers in mind. you could also test out
prefect.yaml
and
prefect deploy
with any new flows you're writing to see how it goes
j

Javier Ochoa

08/30/2023, 6:27 PM
Got it, you gave me the info as I was expecting. Thanks @Nate Yes, I am using deployment from python code due to the flexibility of it (manage variables, env vars, string replacement) which has been difficult for me to achieve the same thing using the
prefect.yaml
file So ok, I will keep using agents until the next version. Thanks!
👍 1
n

Nate

08/30/2023, 6:30 PM
manage variables, env vars, string replacement
this all should be possible in
prefect.yaml
- so when the time comes feel free to ask about any troubles you have with that
j

Javier Ochoa

08/30/2023, 7:00 PM
I tried what it says in here: https://docs.prefect.io/2.11.5/concepts/deployments/?h=runtime#templating-options But it did not work for me. Do you have examples or something?
n

Nate

08/30/2023, 8:58 PM
can you share what you tried and what didn't work (i.e. what error you got?)
j

Javier Ochoa

08/31/2023, 5:38 PM
Sorry @Nate missed the msg: So here is how I am trying:
Copy code
# Welcome to your prefect.yaml file! You can use this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: usg-prefect-2
prefect-version: 2.11.5

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: git@github.com:placeholder/placeholder.git
    branch: dev
    credentials: "{{ prefect.blocks.github-credentials.placeholder }}"

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: water-flow
  version: null
  tags: []
  description: null
  schedule: {}
  flow_name: null
  entrypoint: flows/water-flow.py:main
  parameters: {}
  work_pool:
    name: "{{ $LOCALHOST_ID }}"
    work_queue_name: default
    job_variables: {}
The error says:
Copy code
This deployment configuration references work pool '{{ $LOCALHOST_ID }}' which does not exist. This means no worker will be able to pick up its runs. You can create a work pool in 
the Prefect UI.