https://prefect.io logo
Title
v

Vadym Dytyniak

11/18/2022, 1:53 PM
Hi. We are trying new agent flag released in Prefect 2.6.8 -
prefetch-seconds
. It doesn't work like we expected. Details in the thread.
Agent start command:
prefect agent start --prefetch-seconds 120 -q k8s
a

Anna Geller

11/18/2022, 1:55 PM
before you go on describing here, GitHub issue? looks like a bug
v

Vadym Dytyniak

11/18/2022, 1:56 PM
Flow was scheduled to run at
13:46:00
and
prefetch-seconds=120
. In the logs we see that agent started flow run submission, but pod needed for this flow run was not requested, it requests resources after 120 seconds. Logs:
13:44:00.562 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:05.655 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:10.753 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:15.832 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:20.901 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:26.064 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:31.134 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:36.195 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:41.290 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:46.383 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:51.482 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:56.598 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:01.773 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:06.846 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:11.919 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:16.985 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:22.061 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:27.168 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:32.251 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:37.314 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:42.463 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:47.556 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:52.641 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:57.752 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:46:02.420 | INFO    | prefect.agent - Completed submission of flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:46:02.464 | INFO    | prefect.infrastructure.kubernetes-job - Job 'theta5-klendeth-k-bkl22': Pod has status 'Pending'.
13:47:02.459 | ERROR   | prefect.infrastructure.kubernetes-job - Job 'theta5-klendeth-k-bkl22': Pod never started.
@Anna Geller I will create a bug, but currently I am not sure it is expected behaviour or not. Maybe I misunderstood the feature description. Please clarify.
a

Anna Geller

11/18/2022, 1:58 PM
ahh you're right, it may not be a bug actually
agent started flow run submission, but pod needed for this flow run was not requested
this is only about submission, prefetch means we submit flow run 120 sec earlier than scheduled, no changes to how Prefect requests pods
v

Vadym Dytyniak

11/18/2022, 2:00 PM
Can you explain the difference between flow run submission and pod request?
a

Anna Geller

11/18/2022, 2:01 PM
agent submits flow runs to infra, infra block requests pods - you can check the code for K8s block, the run method may make it more clear
the effect should be: flow run should start 120 sec earlier than it would otherwise
v

Vadym Dytyniak

11/18/2022, 2:07 PM
What is the benefit of this feature? We expected pod will be created and wait for scheduled start time to run flow code as documentation says.
a

Anna Geller

11/18/2022, 2:17 PM
flow run starts 120 sec earlier than scheduled - that's the benefit
v

Vadym Dytyniak

11/18/2022, 2:19 PM
But resources were requested at scheduled start time, not 120 seconds before.
a

Anna Geller

11/18/2022, 2:20 PM
please open an issue with more detailed description, I don't understand the problem here
scheduled time is set based on a deployment schedule, it has nothing to do with infra so I'm confused what you mean
v

Vadym Dytyniak

11/18/2022, 2:25 PM
Imagine that you want to schedule a specific flow to run at 9 AM, but it roughly takes a minute to provision a serverless container. With this new CLI flag, your Prefect agent will begin flow run submission earlier than scheduled (in the example below, 60 seconds earlier) to ensure your container starts on time:
From this this example I understand that serverless container creation should start 60 seconds before schedule time (9 AM). Correct?
a

Anna Geller

11/18/2022, 2:28 PM
yes
v

Vadym Dytyniak

11/18/2022, 2:29 PM
Then why we can't expect that pod should be requested 120 seconds before schedule?
z

Zanie

11/18/2022, 3:11 PM
It's possible the transition to the pending state is blocked until the scheduled time, which would make prefetch not work as intended.
If so, it's a mistake in the prefetch concept and we will investigate a fix.
v

Vadym Dytyniak

11/18/2022, 3:12 PM
Should I create a bug?
z

Zanie

11/18/2022, 3:13 PM
Yes please! I'll look into it today.
v

Vadym Dytyniak

11/18/2022, 3:13 PM
Thanks, will post link here