Camilo Fernandez
07/01/2022, 8:56 AMAnna Geller
07/01/2022, 11:21 AMCamilo Fernandez
07/01/2022, 11:52 AM#values.yaml:
agent:
enabled: "true"
prefectLabels: ["index"]
jobTemplateFilePath: "<s3://prefect-flows/workload.yaml> --log-level DEBUG"
image:
name: "prefecthq/prefect"
tag: "1.2.2-python3.9"
The image is created like this:
#Dockerfile
FROM prefecthq/prefect:1.2.2-python3.9
RUN pip install "prefect[kubernetes]" kubernetes
WORKDIR /
COPY ./init.sh /
# This script does: prefect backend server; this is being executed in each deploy flow run job.
ENTRYPOINT [ "bash", "./init.sh" ]
The flow is stored in a S3 bucket and uses an image I built which is stored in the ECR:
#workload.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: my-k8s-flow
namespace: prefect
spec:
template:
spec:
serviceAccountName: "prefect-server-serviceaccount"
containers:
- name: my-k8s-flow
image: public.ecr.aws/abcd1234/myimage:1.0
imagePullPolicy: Always
restartPolicy: Never
backoffLimit: 4
The agent queues the flow run job properly but the flow runs never change state. The Kubernetes job appears to be completed, but the flow run job doesn't seem to be executed at all. It gets the images from ECR, builts it up to the entrypoint script, it runs it then it completes the Kubernetes job and deletes the pod. The dashboard shows the flow run in a submitted state for a while, the logs show Lazarus rescheduling 3 times then it changes its status to failed.
The agent only logs a completed deployment:
DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Found 1 ready flow run(s): {'0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7'}
DEBUG:agent:Retrieving metadata for 1 flow run(s)...
DEBUG - agent | Found 1 ready flow run(s): {'0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7'}
DEBUG - agent | Retrieving metadata for 1 flow run(s)...
DEBUG:agent:Submitting flow run 0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7 for deployment...
DEBUG - agent | Submitting flow run 0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7 for deployment...
DEBUG:agent:Sleeping flow run poller for 0.25 seconds...
INFO:agent:Deploying flow run 0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7 to execution environment...
DEBUG - agent | Sleeping flow run poller for 0.25 seconds...
INFO - agent | Deploying flow run 0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7 to execution environment...
DEBUG:agent:Updating flow run 0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7 state from Scheduled -> Submitted...
DEBUG - agent | Updating flow run 0b317604-ebf8-490f-b9fb-9cd8ba7e9fa7 state from Scheduled -> Submitted...
DEBUG:agent:Loading job template from '<s3://prefect-flows/workload.yaml>'
DEBUG - agent | Loading job template from '<s3://prefect-flows/workload.yaml>'
DEBUG - agent | Querying for ready flow runs...
DEBUG:agent:Querying for ready flow runs...
The flow should run in a schedule each 5 minutes and print something to stdout:
Something interesting is that logging inside the flow with either Prefect or Python packages logger does not work at all.
#my_flow.py
...
schedule = IntervalSchedule(
start_date=datetime.utcnow() + timedelta(seconds=1),
interval=timedelta(minutes=5),
)
def set_run_config() -> RunConfig:
return KubernetesRun(
job_template_path="<s3://prefect-flows/workload.yaml>",
image="public.ecr.aws/abcd1234/myimage:1.0",
labels=(["index"])
)
def set_storage() -> S3:
return S3(
stored_as_script=True,
key="my_flow.py",
bucket="prefect-flows"
)
@task(state_handlers=[alert_failed], log_stdout=True, slug="test-task")
def test_task():
print('hellooooo ', file=sys.stdout)
with Flow("kubernetes", schedule=schedule, storage=set_storage(),\
run_config=set_run_config()) as flow:
test_task()
# Uncommenting this changes nothing
#flow.run(run_on_schedule=True)
is there anything I'm missing or a way to further debug this problem?
Thanks in advanceAnna Geller
07/01/2022, 12:07 PMlogging inside the flow with either Prefect or Python packages logger does not work at allthis is expected - you can only log in tasks in 1.0 - this is enhanced in 2.0 so that you can log from tasks
Camilo Fernandez
07/01/2022, 12:15 PMAnna Geller
07/01/2022, 12:23 PMCamilo Fernandez
07/01/2022, 12:24 PMAnna Geller
07/01/2022, 12:25 PM