Prefect 2.0.0 Deployment - flow lives in a dedicated Python library/module. I have an internal Python library called

orchester

in which I organize all my orchestration code. In Prefect 1.x the deployment pattern was akin to this:

from orchester.flows import hello_flow
hello_flow.storage = Module(__name__ + ".flows")
hello_flow.register()

In Prefect 2.x-beta I was able to achieve something similar like this:

from orchester.flows import hello_flow
Deployment(
    name="entw",
    flow=hello_flow,
    packager=OrionPackager(serializer=ImportSerializer()),
)
# After that I created the deployment on the local Orion server with the CLI.

Now with Prefect 2.0.0 I am unsure what the deployment pattern would look like. My guess would be to run

prefect deployment build .\orchester\flows\hello_flow.py:hello_flow --name entw

from the cli and to then manually edit the

hello_flow-manifest.json

. I am not sure whether this is the the best approach or whether it needs to be done differently.

{
    "flow_name": "hello_flow",
    "import_path": ".\\orchester\\flows\\hello_flow.py:hello_flow",  # Do I just need to change this line to a module path? 
    "parameter_openapi_schema": {
        "title": "Parameters",
        "type": "object",
        "properties": {
            "name": {
                "title": "name",
                "type": "string"
            }
        },
        "required": [
            "name"
        ],
        "definitions": null
    }
}

TLDR; Does anyone have a good suggestion how to deploy flows that live in their own Python module with Prefect 2.0.0?

The local file system can be used to point to any flow code, including your custom module, so this should work could you share a minimal reproducible example of your flow and module and your deployment? you can share your repo

Thank you @Anna Geller. I'll give it a shot and report back here. Cheers!

in that case you can do

prefect deployment build slackquestion/flows/hello_flow.py:hello_flow --tag xxx --nme yourdeploymentname

@Sebastian did you have any luck with this?

Hi @Anna Geller, @Jack Sundberg - thank you for your help! 🙂 Unfortunately the proposed solution does not work for me (see log at the bottom), since the agent looks for the flow on a local file path (the one where the flow was located when the deployment was created) instead of getting it from a Python module. 😞 To clarify again, I need to reproduce with prefect 2.x what was previously called Module Storage in Prefect 1.x and what was called

ImportSerializer

in 2.x beta. See my original post. My workflow is to: 1. Build the

slackquestion

library containing all flows. 2. Create deployments of the flows in the

slackquestion

library on prefect cloud. 3. Install

slackquestion

library on all agents, docker images etc. 4. Run flows on said agents Perhaps it is something trivial, but I cannot find anything about it in the docs. 😕

Agent started! Looking for work from queue '7dba20ca-76b2-479f-9474-2a0d180d6105'...
17:36:43.506 | INFO    | prefect.agent - Submitting flow run '97481fd1-e2d5-4cba-a12e-7c1957f71224'
17:36:44.204 | INFO    | prefect.infrastructure.process - Opening process 'functional-sawfish'...
17:36:44.230 | INFO    | prefect.agent - Completed submission of flow run '97481fd1-e2d5-4cba-a12e-7c1957f71224'
17:36:48.875 | ERROR   | Flow run 'functional-sawfish' - Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\engine.py", line 247, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\client.py", line 104, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\deployments.py", line 47, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=None, local_path=".")
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\site-packages\prefect\filesystems.py", line 98, in get_directory
    shutil.copytree(from_path, local_path, dirs_exist_ok=True)
  File "C:\Users\dk3\Miniconda3\envs\agent\lib\shutil.py", line 566, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\kwf\\projects\\prefect-deployment'
17:36:50.369 | INFO    | prefect.infrastructure.process - Process 'functional-sawfish' exited cleanly.

Hi @Sebastian, it seems like a lot of people have ran into our issue with python-based deployments & flows stored in modules, so @Anna Geller and the prefect team put out this message: here. Sounds like they're working on it and we need to just sit tight for the time being

Hi @Jack Sundberg thanks a lot for reporting back! Oh wow, fingers crossed it won't be too long.

thanks for your understanding as we go along improving the UX here - there are some complexities in serialization (potential pickling errors depending on python version, portability and import issues with relative paths, payload size, etc) and we work on figuring out the right interface that will allow various deployment patterns in a flexible but still simple and standardized manner it's all in good faith to make it easier and more reliable 🙌