Maikel Penz
12/15/2022, 2:25 AMKubernetesJob
block and apply it to my deployment through Prefect's python API.
When creating the block through the Prefect Cloud UI I can get it to work fine (pull image from ECR and run it on a EKS pod). However, when creating through the Python API my flow fails to start, without any logging. Any ideas?
This is what I am doing
project = "prefect20"
k8s_job_block=KubernetesJob(
image="<AWS-ACCOUNT>.<http://dkr.ecr.eu-west-1.amazonaws.com/kubernetes-job-block:latest|dkr.ecr.eu-west-1.amazonaws.com/kubernetes-job-block:latest>"
)
k8s_job_block.save(project, overwrite=True)
# Create Flow deployment
s3_block_loaded = S3.load(project)
k8s_job_block_loaded = KubernetesJob.load(project)
deployment = Deployment.build_from_flow(
flow=log_flow,
name=project,
storage=s3_block_loaded,
infrastructure=k8s_job_block_loaded,
work_queue_name="dev_queue"
)
deployment.apply()
In my mind it must be credential related but unsure how/where to set them up. If I go to the UI and edit the block (any setting), my flow works.
Something must be done under the hood in the UI.prefect block inspect --id=2585b68d-86e7-4e40-b1e0-292f8b0b560c
Mason Menges
12/15/2022, 9:38 PMMaikel Penz
12/15/2022, 10:42 PM2.7.1
.
When I set the attributes to None
I get error in some of them:
env
none is not an allowed value (type=type_error.none.not_allowed)
labels
none is not an allowed value (type=type_error.none.not_allowed)
job
none is not an allowed value (type=type_error.none.not_allowed)
customizations
none is not an allowed value (type=type_error.none.not_allowed)
pod_watch_timeout_seconds
none is not an allowed value (type=type_error.none.not_allowed)
stream_output
none is not an allowed value (type=type_error.none.not_allowed)
I tried then commenting these ones out, which then allowed me to do the deployment...
k8s_job_block=KubernetesJob(
image="<AWS-ACCOUNT>.<http://dkr.ecr.eu-west-1.amazonaws.com/kubernetes-job-block:latest|dkr.ecr.eu-west-1.amazonaws.com/kubernetes-job-block:latest>",
command=None,
#customizations=None,
#env=None,
image_pull_policy=None,
#job=None,
job_watch_timeout_seconds=None,
#labels=None,
name=None,
namespace=None,
#pod_watch_timeout_seconds=None,
restart_policy=None,
service_account_name=None,
#stream_output=None
)
... but I got the same Submission failed
error when running the flow