Leon Kozlowski
08/26/2022, 7:46 PMDockerStorage
and KubernetesRun
- I have been looking through previous threads and it seems I should be using DockerPackager
, but it looks like a Deployment
doesn’t accept the packager
argument anymore - is there guidance on migrating from a DockerStorage/Kubernetes run workflow to 2.0?Ryan Peden
08/26/2022, 8:31 PM/opt/prefect/flows
but you'll able to provide a different path in the deployment.Leon Kozlowski
08/30/2022, 4:42 PMflow.run_config
, flow.storage
, and optionally flow.schedule
This would look something like this - in a file called deploy.py
"""Prefect Flow Deployment Recipe."""
import os
from datetime import timedelta
from prefect.run_configs import KubernetesRun
from prefect.schedules import IntervalSchedule
from prefect.storage import Docker
from src import config
from src.flow import flow
# Setting flow run config with `prefect.run_config`
flow.run_config = KubernetesRun(
job_template_path=config.KUBE_JOB_TEMPLATE_PATH,
cpu_limit=config.KUBE_CPU_LIMIT,
cpu_request=config.KUBE_CPU_REQUEST,
memory_limit=config.KUBE_MEMORY_LIMIT,
memory_request=config.KUBE_MEMORY_REQUEST,
)
# Setting flow storage with `prefect.storage`
flow.storage = Docker(
env_vars=config.ENVIRONMENT_VARIABLES,
dockerfile="Dockerfile",
image_name=config.DOCKER_IMAGE_NAME,
image_tag=config.DOCKER_IMAGE_TAG,
registry_url=config.DOCKER_REGISTRY_URL,
build_kwargs={
"buildargs": {
"PIP_INDEX_URL": os.getenv("PIP_INDEX_URL"),
},
"nocache": True,
},
)
# Setting flow schedule with `prefect.schedules`
# NOTE: Make sure to import the desired schedule type.
flow.schedule = IntervalSchedule(interval=timedelta(hours=1))
"""Prefect Flow Deployment Recipe."""
import yaml
from prefect.deployments import Deployment
from prefect.packaging import DockerPackager
from prefect.infrastructure import KubernetesJob
from src import config
from src.flow import flow
packager = DockerPackager(
dockerfile="Dockerfile",
registry_url=config.DOCKER_REGISTRY_URL
),
deployment = Deployment(
flow=flow,
name="sandbox",
infrastructure=KubernetesJob(
image="?", # I would rather not pass this as a static value
job=yaml.safe_load(config.KUBE_JOB_TEMPLATE_PATH),
)
)
schedule = "Can only be defined in a deployment.yaml?"
Ryan Peden
08/30/2022, 6:39 PMLeon Kozlowski
08/31/2022, 2:20 PMRyan Peden
08/31/2022, 3:35 PMDockerContainer
and KubernetesJobs
deployments will assume your code is stored in the container if you don't specify external storage for the deployment.
By default, Prefect will look for your code in /opt/prefect/deployment
inside the container, but if you set the path
parameter to a different directory when creating your deployment, Prefect will look for your code in the container at that path instead.Leon Kozlowski
08/31/2022, 6:30 PMDeployment
?KubernetesJob
deployment with the CLI based on what I gatherRyan Peden
08/31/2022, 7:24 PMKubernetesJob
deployment via the CLI, I believe you would need to either edit the deployment YAML, or use a Kubernetes Job block you created and saved via Python or the Orion/Cloud UI