Hi everyone, I realize it's Sunday night after a l...
# prefect-community
j
Hi everyone, I realize it's Sunday night after a long week, so no worries if you wait until tomorrow to respond. I was using orion for a good portion of the beta, and some of the last-minute changes to Deployments are confusing me. For starters, I would dynamically create deployments like this:
Copy code
deployment = Deployment(
    name="example-deploy",
    flow=my_flow_object,  # defined elsewhere
    packager=OrionPackager(serializer=PickleSerializer()),
    tags=[
        "some-tag1",
        "some-tag2",
    ],
)

deployment_id = await deployment.create()
But this code is now broken and the docs recommend sticking to yaml files. How would I go about updating this? Can we still specify a
packager
and is it possible to create a deployment in python without creating a yaml file?
2
👀 2
k
Hi Jack,
Deployment
is no longer supported in 2.0. I recommend that you run the following:
Copy code
prefect deployment build name-to-your-flow:my-flow-object -t "some-tag1" -t "some-tag2"
at the directory that contains your flow and its imports
j
thanks for the reply! My flows are created dynamically and also within an installed package -- so there's no file/directory for me to point to. So this command actually won't work for me. Will the deployment class be fully removed in the future? And is there no way to create a deployment without a flow file?
For example, I access my prefect flows like this:
Copy code
from my_package.workflows import example
prefect_flow = example.to_prefect_flow()
a
Do you run it in a local process? Importing via a module vs specifying path on a manifest is not much different
o
Hi! I was also going to ask the same question as Jack. With the way I'm wanting to create deployments, it would be very useful to create them directly in python. Is there any way it can still be done? Is it a case of using
OrionClient.create_deployment
or is there a more efficient way?
a
all blocks can be created in Python, e.g.:
Copy code
from prefect.infrastructure import DockerContainer

infra_block = DockerContainer()
infra_block.save("docker")
same for file system
Copy code
from prefect.filesystems import S3

block = S3()
block.save("dev")
one you have your blocks and flows, you can even run a subprocess to build and apply we anticipate most users will be creating deployments from CI/CD after a PR review and merge, that's why it's optimized for CLI/API-driven UX
o
Sorry to ask again but is there an easy way to do this in the context of deployments?
j
Hi Anna, thanks for the follow-up! And Owen, it's good to hear I'm not alone haha. I imagine most users will create deployments from CI/CD when building private & self-contained flows -- but I suspect many people in the open source community are starting to build prefect workflow libraries that can be installed as a python package which handles the creation of deployments itself. For example, I do this in my repo already and have internal methods that build on prefect deployments (see a quick code snippet here). So during the beta, I found the packaging and deployments modules super useful -- and am surprised deployments using
Deployment
and
PickleSerializer
are now staged for removal. I may be confused on the new yaml approach, but it still seems like a single file is required for deployments. Seems like a major restriction. PS -- yes, I often run in a local process, but I build in the option for my users to switch where runs complete if they'd like to use Docker etc..
1
@Owen Cook you might find the code snippet useful. The
Deployment.create()
method contains a higher-level call to
client.create_deployment
though the method currently contains a bug and it's depreciated 😅
@Anna Geller I'm so sorry to keep bothering you, but I'm in a panic because of this feature removal. Not having python-based deployments breaks a lot of my core features. Is there anything I can do in the meantime? Maybe even just open a request for this feature to be added to the backlog?
o
Nice one, cheers 🙂 Been having a little look, not had too much luck yet. With the way everything worked before I didn't have to think about storage or infrastructure so will probably help my understanding. The python deployment create feature was very useful though and seemed much simpler. I think it will mean it's a bit harder to set things up for my use case.
🚀 1
@Jack Sundberg, just to get something working, I came up with this example. Not sure how far you got looking at how your deployments are working but this might be helpful so thought I'd share.
Copy code
"""Create deployment using prefect api."""
import asyncio
import json
import time
from pathlib import Path

from prefect import Manifest, flow, get_client
from prefect.client import OrionClient
from prefect.deployments import FlowScript
from prefect.filesystems import LocalFileSystem
from prefect.flows import Flow, load_flow_from_script
from prefect.infrastructure import Process
from prefect.packaging.orion import OrionPackager
from prefect.utilities.callables import parameter_schema


@flow
def example_flow(num: int) -> None:
    """Example flow."""
    for i in range(10):
        print(i + 1)
        time.sleep(0.1)
    print(f"Input number: {num}")


def flow_from_flow_script(flow_script: FlowScript) -> Flow:
    """Load flow from a flow script.

    Args:
        flow_script (FlowScript): Flow script to be turned into flow.

    Returns:
        Flow: flow to be produced from flow script.
    """
    return load_flow_from_script(
        flow_script.path.expanduser().resolve(), flow_name=flow_script.name
    )


async def create_deployment() -> None:
    """Create a deployment."""

    client: OrionClient = get_client()

    # set the flow
    my_flow = flow_from_flow_script(
        FlowScript(
            path=Path(__file__).parent / "deployment.py",
            name="example-flow",
        )
    )

    # create manifest from flow (OrionPackager)
    manifest = await OrionPackager().package(my_flow)

    # create flow and get flow id
    flow_id = await client.create_flow(my_flow)

    # create infrastructure
    updates = {}
    if "image" in manifest.__fields__:
        updates["image"] = manifest.image

    infrastructure = Process()
    infrastructure = infrastructure.copy(update=updates)
    infrastructure_id = await infrastructure._save(is_anonymous=True)

    # set up storage
    storage = LocalFileSystem(basepath=Path(__file__).parent.absolute())
    storage_id = await storage._save(is_anonymous=True)

    # set up parameters
    params = {"num": 5}
    openapi_schema = parameter_schema(my_flow)

    # get the manifest
    manifest_path = f"{my_flow.fn.__name__}-manifest.json"
    manifest = Manifest(
        flow_name=my_flow.name,
        import_path=f"{Path(__file__)}:{my_flow.fn.__name__}",
        parameter_openapi_schema=openapi_schema,
    )
    with open(manifest_path, "w", encoding="UTF-8") as manifest_file:
        json.dump(manifest.dict(), manifest_file, indent=4)

    # create deployment
    return await client.create_deployment(
        flow_id=flow_id,
        name=my_flow.name,
        parameters=params,
        description="Some description.",
        tags=["test"],
        manifest_path=manifest_path,
        storage_document_id=storage_id,
        infrastructure_document_id=infrastructure_id,
        parameter_openapi_schema=openapi_schema,
    )


asyncio.run(create_deployment())
j
Thanks @Owen Cook! I'll give this a try. Looks like a great approach and will save me a bunch of time. Thank you so much 😄 Also before you sent this, I started a github discussion on this topic here. Feel free to post your example there for others to see. I've also reached out to the prefect team about paid support, so hopefully we can get an officially supported fix on top of this example.
after going through this some more, I'll probably have to keep working to update your example. For example, make sure you use the client within a context like
async with get_client() as client:
. Also I need to make sure the solution also doesn't depend on a single file script (e.g.
deployment.py
) with the flow in it.
o
@Jack Sundberg, cool I'll have a look at that! Ah nice one, that would be great. Cheers, I wasn't aware of that being best practice, I'll have to give everything an update -- looking at maybe trying to create a class to handle some more deployment creation stuff. Yeah that was quite a big limitation with what I wrote but just needed something simple to see if I could get things working 😄
a
@Jack Sundberg we are working on further improving the deployment UX and making it clearer how to apply it to various use cases and deployment patterns/environments; I'd recommend waiting for official recipes coming in the next days and weeks before committing to a deployment story just yet.
e.g. FlowScript is deprecated in 2.0.1 and I would recommend using file system blocks rather than packagers for packaging flow code - have you read the new Deployment documentation released as part of GA? https://docs.prefect.io/concepts/deployments/
j
yep, I read through the quick-start, concepts docs, api docs, and source code for deployments -- my use case just isn't covered in the docs yet. Thanks for the extra info too! I'll also respond back over in the github discussion. 😄
🎯 1
🙌 1
🙏 1