https://prefect.io logo
l

Leon Kozlowski

08/26/2022, 7:46 PM
I am trying to migration from a 0.15.12 setup using
DockerStorage
and
KubernetesRun
- I have been looking through previous threads and it seems I should be using
DockerPackager
, but it looks like a
Deployment
doesn’t accept the
packager
argument anymore - is there guidance on migrating from a DockerStorage/Kubernetes run workflow to 2.0?
The DockerPackager also generates the image name at runtime, but do not see a way to pass a tag
r

Ryan Peden

08/26/2022, 8:31 PM
I believe this PR will cover what you need when it is merged: https://github.com/PrefectHQ/prefect/pull/6574 If it gets merged as-is, you'd build your flow code into a Docker image, then create a KubernetesJob deployment without any remote storage specified. By default, Prefect will look for your flow code in
/opt/prefect/flows
but you'll able to provide a different path in the deployment.
l

Leon Kozlowski

08/30/2022, 4:42 PM
I’m not exactly following, do you suggest building the docker image and pushing to registry separately from a flow deployment?
I am trying to replicate the following configuration from prefect 0.15.11 Previously, the deployment would consist of 2 or 3 components:
flow.run_config
,
flow.storage
, and optionally
flow.schedule
This would look something like this - in a file called
deploy.py
Copy code
"""Prefect Flow Deployment Recipe."""
import os
from datetime import timedelta

from prefect.run_configs import KubernetesRun
from prefect.schedules import IntervalSchedule
from prefect.storage import Docker

from src import config
from src.flow import flow

# Setting flow run config with `prefect.run_config`
flow.run_config = KubernetesRun(
    job_template_path=config.KUBE_JOB_TEMPLATE_PATH,
    cpu_limit=config.KUBE_CPU_LIMIT,
    cpu_request=config.KUBE_CPU_REQUEST,
    memory_limit=config.KUBE_MEMORY_LIMIT,
    memory_request=config.KUBE_MEMORY_REQUEST,
)

# Setting flow storage with `prefect.storage`
flow.storage = Docker(
    env_vars=config.ENVIRONMENT_VARIABLES,
    dockerfile="Dockerfile",
    image_name=config.DOCKER_IMAGE_NAME,
    image_tag=config.DOCKER_IMAGE_TAG,
    registry_url=config.DOCKER_REGISTRY_URL,
    build_kwargs={
        "buildargs": {
            "PIP_INDEX_URL": os.getenv("PIP_INDEX_URL"),
        },
        "nocache": True,
    },
)

# Setting flow schedule with `prefect.schedules`
# NOTE: Make sure to import the desired schedule type.
flow.schedule = IntervalSchedule(interval=timedelta(hours=1))
I can’t find any docs on how to replicate this in 2.0 - This is my working example
Copy code
"""Prefect Flow Deployment Recipe."""
import yaml

from prefect.deployments import Deployment
from prefect.packaging import DockerPackager
from prefect.infrastructure import KubernetesJob

from src import config
from src.flow import flow

packager = DockerPackager(
    dockerfile="Dockerfile",
    registry_url=config.DOCKER_REGISTRY_URL
),

deployment = Deployment(
    flow=flow,
    name="sandbox",
    infrastructure=KubernetesJob(
        image="?",  # I would rather not pass this as a static value
        job=yaml.safe_load(config.KUBE_JOB_TEMPLATE_PATH),
    )
)

schedule = "Can only be defined in a deployment.yaml?"
r

Ryan Peden

08/30/2022, 6:39 PM
Thanks for the additional information, Leon. I'm going to check with some of my colleagues and then get back to you with an answer
🙏 1
Hi Leon, apologies for the slow reply. The recommended way to do this is, as you mentioned previously, to build the image and push it to a registry separately from the deployment. I believe you can do it in the same file as your deployment, however, if you use the Docker SDK for Python, specifically the Image build and push methods.
l

Leon Kozlowski

08/31/2022, 2:20 PM
So docker storage is no longer natively supported for flow deployments?
r

Ryan Peden

08/31/2022, 3:35 PM
The way I'd describe it is: Docker storage is natively supported in that
DockerContainer
and
KubernetesJobs
deployments will assume your code is stored in the container if you don't specify external storage for the deployment. By default, Prefect will look for your code in
/opt/prefect/deployment
inside the container, but if you set the
path
parameter to a different directory when creating your deployment, Prefect will look for your code in the container at that path instead.
This was just added in yesterday's 2.3.0 release
l

Leon Kozlowski

08/31/2022, 6:30 PM
It is still a requirement to manually build the docker image independent of a
Deployment
?
And it is not possible to use a custom
KubernetesJob
deployment with the CLI based on what I gather
r

Ryan Peden

08/31/2022, 7:24 PM
yes, you would need to build the Docker image and push it to a registry (using either the Docker Python API or Docker CLI) For a customized
KubernetesJob
deployment via the CLI, I believe you would need to either edit the deployment YAML, or use a Kubernetes Job block you created and saved via Python or the Orion/Cloud UI
4 Views