how might I make use of a serializer to package the module my `flow.py` is contained in?

Kevin Grismore

07/12/2022, 9:26 PM

how might I make use of a serializer to package the module my

flow.py

is contained in?

✅ 1

Kevin Grismore

07/12/2022, 9:28 PM

here is my `quotes_deployment.py`:

Copy code

from prefect.deployments import Deployment
from prefect.deployments import FlowScript
from prefect.flow_runners import KubernetesFlowRunner
from prefect.packaging import DockerPackager
from prefect.software import CondaEnvironment
from pathlib import Path

Deployment(
    name="quotes_spider",
    flow=FlowScript(
        path=Path(__file__).parent / "flow.py",
        name="quotes",
    ),
    flow_runner=KubernetesFlowRunner,
    packager=DockerPackager(python_environment=CondaEnvironment().from_environment),
    tags=['k8s','local']
)

Kevin Grismore

07/12/2022, 9:39 PM

to be clear, my goal is to: 1. create a docker image with my conda environment (and place it in a container registry, which I left out here) 2. package my flow with the module it's in and store it

Kevin Grismore

07/12/2022, 9:41 PM

maybe that's not possible though? It may make more sense to create that docker image and register it separately, then package my module and just ask for the image in my kubernetes flow runner args

Anna Geller

07/13/2022, 10:42 AM

try this syntax:

Copy code

Deployment(
    flow=hello,
    name="docker_packager",
    packager=DockerPackager(
        base_image="prefecthq/prefect:dev-python3.9",
        python_environment=PythonEnvironment(
            python_version="3.9",
            pip_requirements=["requests==2.28.0"],
        ),
    ),
    flow_runner=DockerFlowRunner(),
)

pushing to a registry doesn't work yet atm

5 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.