Joe B
09/05/2023, 1:17 PMNate
09/05/2023, 2:16 PMprefect.yaml
at the root of your repoJohn Kang
09/05/2023, 2:26 PMJoe B
09/05/2023, 2:29 PMWorker 'ProcessWorker bc574b38-164d-423d-9fa9-3c78b6965e51' submitting flow run 'a8df70ee-63e7-4119-b9e7-0875ab4b4bb9'
10:37:36 AM
prefect.flow_runs.worker
Opening process...
10:37:36 AM
prefect.flow_runs.worker
Process running command: /usr/local/bin/python -m prefect.engine in /tmp/tmpspe8bqgqprefect
10:37:36 AM
prefect.flow_runs.worker
Completed submission of flow run 'a8df70ee-63e7-4119-b9e7-0875ab4b4bb9'
10:37:36 AM
prefect.flow_runs.worker
Running 1 deployment pull steps
10:37:37 AM
prefect.flow_runs
Changing working directory to '/opt/prefect'
10:37:37 AM
prefect.flow_runs
Importing flow code from 'flows/ingestion.py:ingestion'
10:37:37 AM
prefect.flow_runs
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 936, in exec_module
File "<frozen importlib._bootstrap_external>", line 1073, in get_code
File "<frozen importlib._bootstrap_external>", line 1130, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/flows/ingestion.py'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 395, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/deployments/deployments.py", line 222, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/flows.py", line 975, in load_flow_from_entrypoint
flow = import_object(entrypoint)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 201, in import_object
module = load_script_as_module(script_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/importtools.py", line 164, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'flows/ingestion.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
10:37:37 AM
prefect.flow_runs
Process 27 exited cleanly.
prefect init
to get a prefect.yaml
file and see what may or may not changeJohn Kang
09/05/2023, 3:48 PM# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/johnkangw/site_analytics.git>
# TODO: Update branch to master or main
branch: kubernetes-testing
access_token: '{{ prefect.blocks.secret.deployment-kubernetes-worker-weather-process-comp-files-repo-token
}}'
Joe B
09/05/2023, 3:52 PMNate
09/05/2023, 4:04 PMMy bash script is managing the deployment build and I think I need to hand some of that responsibility over to the yaml file.yes, in the
deployments
section of the prefect.yaml
you can template (and/or hardcode) in the stuff you would have passed into prefect deployment build
(more or less)
the important difference(s) being that you no longer need to specify infra / storage block
• no more storage block required for deployment -> instead use a pull
step to tell the worker where to get the code
• no more infra block required for deployment -> instead the work pool assigned to your deployment tells the worker how to execute the flow run