Keith
09/26/2022, 4:56 PMfrom prefect2_flow import extract_load_transform
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob
from prefect.orion.schemas.schedules import CronSchedule
schedule = CronSchedule(cron="15 * * * *", timezone="UTC")
deployment = Deployment().build_from_flow(
flow=extract_load_transform,
name="test_hourly_elt",
parameters={
'collection_duration': '1h',
'uri': '<https://google.com>',
},
skip_upload=True,
schedule=schedule,
tags=["test"],
version=2,
work_queue_name="test-kubernetes",
infrastructure=KubernetesJob(
finished_job_ttl=30,
image="us-central1-docker.pkg.dev/.../prefect2-flows/elt:latest",
image_pull_policy="Always",
namespace="test-prefect2",
pod_watch_timeout_seconds=180,
job={
"apiVersion": "batch/v1",
"kind": "Job",
"metadata": {"labels": {}},
"spec": {
"template": {
"spec": {
"parallelism": 1,
"completions": 1,
"restartPolicy": "Never",
"containers": [
{
"name": "prefect-job",
"env": [],
"resources": {
"requests": {
"memory": "5Gi",
"cpu": "2",
"ephemeral-storage": "1Gi",
}
}
}
],
}
}
},
}
)
)
if __name__ == "__main__":
deployment.apply()
1. Is it okay to set skip_upload
to true? Since we are using Docker I'm not sure what the benefit of uploading our project to GCS is.
2. The Prefect API has a parameter for is_schedule_active
but it doesn't look like that parameter has made it over to the Python API b/c when I try to add it above it gives me an error about not including more parameters than needed, is this something I can contribute to or add functionality for?
3. The k8s job configuration is pretty verbose to request resources above the default, is there a better way to set these that I am missing?
Thank you for the time and any help provided!Christopher Boyd
09/27/2022, 3:19 PMfrom prefect.infrastructure import KubernetesJob
k8s_job=KubernetesJob(
namespace="prefect2",
image="prefecthq/prefect:2.3.0-python3.9",
job=KubernetesJob.job_from_file("modified_run_job.yaml")
)
k8s_job.save("k8sdev")
customizations=[
{
"op": "add",
"path": "/spec/imagePullSecrets",
"value": [{'name': 'dockerhub'}],
},
],
or
customizations=[
{
"op": "add",
"path": "/spec/imagePullSecrets",
"value": [{'name': 'dockerhub'}],
},
{
"op": "add",
"path": "/spec/template/spec/resources",
"value": {"limits": {"memory": "8Gi", "cpu": "4000m"}},
}
],
Keith
09/27/2022, 6:41 PMis_schedule_active
, here are a couple of references to the actual flag and some test deployments:
• Flag toggling - https://github.com/PrefectHQ/prefect/blob/main/src/prefect/orion/api/deployments.py#L232
• Part of CLI response - https://github.com/PrefectHQ/prefect/blob/main/docs/concepts/deployments.md?plain=1#L449
• API create deployment with is_schedule_active
flag turned off - https://github.com/PrefectHQ/prefect/blob/main/tests/orion/models/test_deployments.py#L128Christopher Boyd
09/27/2022, 6:43 PMDekel R
11/20/2022, 10:27 AM