https://prefect.io logo
Title
s

Sebastian

07/29/2022, 2:27 PM
Prefect 2.0.0 Deployment - flow lives in a dedicated Python library/module. I have an internal Python library called
orchester
in which I organize all my orchestration code. In Prefect 1.x the deployment pattern was akin to this:
from orchester.flows import hello_flow
hello_flow.storage = Module(__name__ + ".flows")
hello_flow.register()
In Prefect 2.x-beta I was able to achieve something similar like this:
from orchester.flows import hello_flow
Deployment(
    name="entw",
    flow=hello_flow,
    packager=OrionPackager(serializer=ImportSerializer()),
)
# After that I created the deployment on the local Orion server with the CLI.
Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run
prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw
from the cli and to then manually edit the
hello_flow-manifest.json
. I am not sure whether this is the the best approach or whether it needs to be done differently.
{
    "flow_name": "hello_flow",
    "import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow",  # Do I just need to change this line to a module path? 
    "parameter_openapi_schema": {
        "title": "Parameters",
        "type": "object",
        "properties": {
            "name": {
                "title": "name",
                "type": "string"
            }
        },
        "required": [
            "name"
        ],
        "definitions": null
    }
}
TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?
1
👀 4
a

Anna Geller

07/29/2022, 2:44 PM
The local file system can be used to point to any flow code, including your custom module, so this should work could you share a minimal reproducible example of your flow and module and your deployment? you can share your repo
🙏 1
s

Sebastian

07/29/2022, 2:48 PM
Thank you @Anna Geller. I'll give it a shot and report back here. Cheers!
🙌 1
@Anna Geller https://gitlab.com/public-questions/prefect-deployment/-/tree/main Is this sufficient? The idea would be that my execution environment just needs to install the
slackquestion
library and will then have all flows and their dependencies. As previously mentioned I was able to do this with the orion beta with
[...]packager=OrionPackager(serializer=ImportSerializer()) [...]
a

Anna Geller

07/30/2022, 7:15 PM
in that case you can do
prefect deployment build slackquestion/flows/hello_flow.py:hello_flow --tag xxx --nme yourdeploymentname
this will use by defualt process infra block and local filesystem block
everything runs locally then, no need for packagers and serializers
j

Jack Sundberg

08/02/2022, 2:51 PM
@Sebastian did you have any luck with this?
s

Sebastian

08/03/2022, 3:53 PM
Hi @Anna Geller, @Jack Sundberg - thank you for your help! 🙂 Unfortunately the proposed solution does not work for me (see log at the bottom), since the agent looks for the flow on a local file path (the one where the flow was located when the deployment was created) instead of getting it from a Python module. 😞 To clarify again, I need to reproduce with prefect 2.x what was previously called Module Storage in Prefect 1.x and what was called
ImportSerializer
in 2.x beta. See my original post. My workflow is to: 1. Build the
slackquestion
library containing all flows. 2. Create deployments of the flows in the
slackquestion
library on prefect cloud. 3. Install
slackquestion
library on all agents, docker images etc. 4. Run flows on said agents Perhaps it is something trivial, but I cannot find anything about it in the docs. 😕
Agent started! Looking for work from queue '7dba20ca-76b2-479f-9474-2a0d180d6105'...
17:36:43.506 | INFO    | prefect.agent - Submitting flow run '97481fd1-e2d5-4cba-a12e-7c1957f71224'
17:36:44.204 | INFO    | prefect.infrastructure.process - Opening process 'functional-sawfish'...
17:36:44.230 | INFO    | prefect.agent - Completed submission of flow run '97481fd1-e2d5-4cba-a12e-7c1957f71224'
17:36:48.875 | ERROR   | Flow run 'functional-sawfish' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\deployments.py", line 47, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\filesystems.py", line 98, in get_directory
    shutil.copytree(from_path, local_path, dirs_exist_ok=True)
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\shutil.py", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\kwf\\projects\\prefect-deployment'
17:36:50.369 | INFO    | prefect.infrastructure.process - Process 'functional-sawfish' exited cleanly.
j

Jack Sundberg

08/03/2022, 4:02 PM
Hi @Sebastian, it seems like a lot of people have ran into our issue with python-based deployments & flows stored in modules, so @Anna Geller and the prefect team put out this message: here. Sounds like they're working on it and we need to just sit tight for the time being
:thank-you: 1
🙌 2
s

Sebastian

08/03/2022, 4:09 PM
Hi @Jack Sundberg thanks a lot for reporting back! Oh wow, fingers crossed it won't be too long.
a

Anna Geller

08/03/2022, 4:41 PM
thanks for your understanding as we go along improving the UX here - there are some complexities in serialization (potential pickling errors depending on python version, portability and import issues with relative paths, payload size, etc) and we work on figuring out the right interface that will allow various deployment patterns in a flexible but still simple and standardized manner it's all in good faith to make it easier and more reliable 🙌
❤️ 3