Abhinav Chordia
04/14/2023, 7:43 PMChris White
04/17/2023, 7:42 PMAbhinav Chordia
04/17/2023, 7:44 PMChris White
04/17/2023, 7:46 PMprefect worker start -t kubernetes -p new-k8s-work-pool
and then use prefect project init --recipe docker
to setup a dockerized deployment; push the deployment to the new-k8s-work-pool
and that should be enough to get your first runAbhinav Chordia
04/17/2023, 7:57 PMdescription: null
entrypoint: null
flow_name: null
name: null
parameters: {}
schedule: null
tags: []
version: null
work_pool:
job_variables:
image: '{{ image_name }}'
name: null
work_queue_name: null
This is my current prefect.yaml
# File for configuring project / deployment build, push and pull steps
# Generic metadata about this project
name: hypo
prefect-version: 2.10.3
# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.1.0
image_name: <http://docker.gh.st/hypo|docker.gh.st/hypo>
tag: dev
dockerfile: Dockerfile
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.set_working_directory:
directory: /opt/prefect/hypo
from hypo.prefect_apps.example.example_flow import call_api
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob, KubernetesImagePullPolicy
from prefect.filesystems import Azure
from prefect.blocks.kubernetes import KubernetesClusterConfig
customizations = [
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {
"requests": {
"cpu": "2000m",
"memory": "4Gi"
},
"limits": {
"cpu": "4000m",
"memory": "8Gi",
}
},
}
]
k8s_job = KubernetesJob(
namespace="prefect",
image="<http://docker.gh.st/hypo:dev|docker.gh.st/hypo:dev>",
image_pull_policy=KubernetesImagePullPolicy.ALWAYS,
finished_job_ttl=300,
job_watch_timeout_seconds=600,
pod_watch_timeout_seconds=600,
service_account_name="prefect-server",
customizations=customizations,
)
k8s_job.save("devk8s", overwrite=True)
# az_block = Azure.load("ghprodstorage1") # load a pre-defined block
kubernetes_job_block = KubernetesJob.load("devk8s")
deployment = Deployment.build_from_flow(
flow=call_api,
name="example",
version=1,
infrastructure=kubernetes_job_block,
work_queue_name="default",
path="/user/abhinav",
# storage=az_block
)
deployment.apply()
Chris White
04/17/2023, 8:38 PMAbhinav Chordia
04/17/2023, 8:39 PM# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
requires: prefect-docker>0.1.0
image_name: <http://docker.gh.st/hypo|docker.gh.st/hypo>
tag: dev
dockerfile: Dockerfile
# push section allows you to manage if and how this project is uploaded to remote locations
push: null
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.set_working_directory:
directory: /opt/prefect/hypo
Chris White
04/17/2023, 8:43 PM"{{ image_name }}"
in your deployment.yaml
so if you use the prefect deploy
CLI you should be good go toAbhinav Chordia
04/17/2023, 8:53 PMChris White
04/17/2023, 8:57 PMprefect.yaml
file that contains defaults for the build / push / pull steps and could override these on a per-deployment basis.
Also while these core steps provide us with robust building blocks (the explicit nature of the steps is incredibly convenient for extending and debugging compared to the previous blocks-based implementation), there's still a lot we can do to simplify the UX around deployments from this, so an SDK isn't totally out of the question if we can collect enough feedback about the use cases it would solve for.Abhinav Chordia
04/17/2023, 9:07 PMChris White
04/17/2023, 9:09 PMAbhinav Chordia
04/17/2023, 9:10 PMChris White
04/17/2023, 9:11 PM