Sebastian
07/29/2022, 2:27 PMorchester
in which I organize all my orchestration code.
In Prefect 1.x the deployment pattern was akin to this:
from orchester.flows import hello_flow
hello_flow.storage = Module(__name__ + ".flows")
hello_flow.register()
In Prefect 2.x-beta I was able to achieve something similar like this:
from orchester.flows import hello_flow
Deployment(
name="entw",
flow=hello_flow,
packager=OrionPackager(serializer=ImportSerializer()),
)
# After that I created the deployment on the local Orion server with the CLI.
Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw
from the cli and to then manually edit the hello_flow-manifest.json
. I am not sure whether this is the the best approach or whether it needs to be done differently.
{
"flow_name": "hello_flow",
"import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow", # Do I just need to change this line to a module path?
"parameter_openapi_schema": {
"title": "Parameters",
"type": "object",
"properties": {
"name": {
"title": "name",
"type": "string"
}
},
"required": [
"name"
],
"definitions": null
}
}
TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?Anna Geller
07/29/2022, 2:44 PMSebastian
07/29/2022, 2:48 PMslackquestion
library and will then have all flows and their dependencies.
As previously mentioned I was able to do this with the orion beta with [...]packager=OrionPackager(serializer=ImportSerializer()) [...]
Anna Geller
07/30/2022, 7:15 PMprefect deployment build slackquestion/flows/hello_flow.py:hello_flow --tag xxx --nme yourdeploymentname
Jack Sundberg
08/02/2022, 2:51 PMSebastian
08/03/2022, 3:53 PMImportSerializer
in 2.x beta. See my original post.
My workflow is to:
1. Build the slackquestion
library containing all flows.
2. Create deployments of the flows in the slackquestion
library on prefect cloud.
3. Install slackquestion
library on all agents, docker images etc.
4. Run flows on said agents
Perhaps it is something trivial, but I cannot find anything about it in the docs. 😕
Agent started! Looking for work from queue '7dba20ca-76b2-479f-9474-2a0d180d6105'...
17:36:43.506 | INFO | prefect.agent - Submitting flow run '97481fd1-e2d5-4cba-a12e-7c1957f71224'
17:36:44.204 | INFO | prefect.infrastructure.process - Opening process 'functional-sawfish'...
17:36:44.230 | INFO | prefect.agent - Completed submission of flow run '97481fd1-e2d5-4cba-a12e-7c1957f71224'
17:36:48.875 | ERROR | Flow run 'functional-sawfish' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\client.py", line 104, in with_injected_client
return await fn(*args, **kwargs)
File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\deployments.py", line 47, in load_flow_from_flow_run
await storage_block.get_directory(from_path=None, local_path=".")
File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\filesystems.py", line 98, in get_directory
shutil.copytree(from_path, local_path, dirs_exist_ok=True)
File "C:\Users\dk3\Miniconda3\envs\agent\lib\shutil.py", line 566, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\kwf\\projects\\prefect-deployment'
17:36:50.369 | INFO | prefect.infrastructure.process - Process 'functional-sawfish' exited cleanly.
Jack Sundberg
08/03/2022, 4:02 PMSebastian
08/03/2022, 4:09 PMAnna Geller
08/03/2022, 4:41 PM