https://prefect.io
Join Slack
Hello prefect friends, I have an application that uses agents and I'm working on refactoring to leve...
# ask-community
j
Hello prefect friends, I have an application that uses agents and I'm working on refactoring to leverage prefect workers. The app is driven by docker / docker-compose with one service dedicated to agents. Flows are deployed using the "prefect deployment build " command in a bash script kicked off when the docker compose is brought up. I've updated all deployment commands to "prefect --no-prompt deploy", and changed the docker compose entry point from prefect agent start -q <queue_name> to prefect worker start -p <pool_name>. However when I attempt to run a flow I get an error message of "file not found". Am I missing something obvious in my set up? I can share more info as needed. Thanks in advance!
n
hi @Joe B - do you have the trace? im wondering which file it cant find. you'll need to have your 
prefect.yaml
at the root of your repo
j
Does it have to do with where you are storing your code?
Is it on Github, GCS, AWS, etc.?
Maybe it is not uploading correctly
j
Hi @Nate, @John Kang, thanks for the responses. I reverted my changes and am working on getting that stack trace back. The prefect.yaml was good info, I don't think I was properly generating one. My code is currently local for the initial refactor.
👍 1
Here is the trace:
Copy code
Worker 'ProcessWorker bc574b38-164d-423d-9fa9-3c78b6965e51' submitting flow run 'a8df70ee-63e7-4119-b9e7-0875ab4b4bb9'
10:37:36 AM
prefect.flow_runs.worker
Opening process...
10:37:36 AM
prefect.flow_runs.worker
Process running command: /usr/local/bin/python -m prefect.engine in /tmp/tmpspe8bqgqprefect
10:37:36 AM
prefect.flow_runs.worker
Completed submission of flow run 'a8df70ee-63e7-4119-b9e7-0875ab4b4bb9'
10:37:36 AM
prefect.flow_runs.worker
Running 1 deployment pull steps
10:37:37 AM
prefect.flow_runs
Changing working directory to '/opt/prefect'
10:37:37 AM
prefect.flow_runs
Importing flow code from 'flows/ingestion.py:ingestion'
10:37:37 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 936, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1073, in get_code
  File "<frozen importlib._bootstrap_external>", line 1130, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows/ingestion.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 222, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/flows.py", line 975, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 201, in import_object
    module = load_script_as_module(script_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/ingestion.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
10:37:37 AM
prefect.flow_runs
Process 27 exited cleanly.
I'm going to attempt to run 
prefect init
to get a 
prefect.yaml
file and see what may or may not change
Still erroring out - I think migrating from agents to workers will be a heavier lift than I initially expected. My bash script is managing the deployment build and I think I need to hand some of that responsibility over to the yaml file.
j
When I converted over from agents to workers it was a bit of a lift too
🎯 1
how are you setting your remote code?
I set it up like this in the yaml:
Copy code
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/johnkangw/site_analytics.git>
    # TODO: Update branch to master or main
    branch: kubernetes-testing
    access_token: '{{ prefect.blocks.secret.deployment-kubernetes-worker-weather-process-comp-files-repo-token
      }}'
It was key for me to setup the access_token otherwise I couldn't pull the code from github
not sure how you are pulling your code
j
Thanks John, this is good info. I'm attempting to run things locally right now but will be pulling from github when things start shaping up
n
My bash script is managing the deployment build and I think I need to hand some of that responsibility over to the yaml file.
yes, in the 
deployments
section of the 
prefect.yaml
you can template (and/or hardcode) in the stuff you would have passed into 
prefect deployment build
(more or less) the important difference(s) being that you no longer need to specify infra / storage block • no more storage block required for deployment -> instead use a 
pull
step to tell the worker where to get the code • no more infra block required for deployment -> instead the work pool assigned to your deployment tells the worker how to execute the flow run
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by