https://prefect.io logo
Join Slack
Powered by
Hi. We are trying new agent flag released in Prefe...
# prefect-community
v
Hi. We are trying new agent flag released in Prefect 2.6.8 - 
prefetch-seconds
. It doesn't work like we expected. Details in the thread.
Agent start command: 
prefect agent start --prefetch-seconds 120 -q k8s
a
before you go on describing here, GitHub issue? looks like a bug
v
Flow was scheduled to run at 
13:46:00
and 
prefetch-seconds=120
. In the logs we see that agent started flow run submission, but pod needed for this flow run was not requested, it requests resources after 120 seconds. Logs:
Copy code
13:44:00.562 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:05.655 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:10.753 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:15.832 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:20.901 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:26.064 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:31.134 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:36.195 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:41.290 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:46.383 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:51.482 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:56.598 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:01.773 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:06.846 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:11.919 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:16.985 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:22.061 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:27.168 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:32.251 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:37.314 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:42.463 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:47.556 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:52.641 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:57.752 | INFO    | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:46:02.420 | INFO    | prefect.agent - Completed submission of flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:46:02.464 | INFO    | prefect.infrastructure.kubernetes-job - Job 'theta5-klendeth-k-bkl22': Pod has status 'Pending'.
13:47:02.459 | ERROR   | prefect.infrastructure.kubernetes-job - Job 'theta5-klendeth-k-bkl22': Pod never started.
@Anna Geller I will create a bug, but currently I am not sure it is expected behaviour or not. Maybe I misunderstood the feature description. Please clarify.
a
ahh you're right, it may not be a bug actually
agent started flow run submission, but pod needed for this flow run was not requested
this is only about submission, prefetch means we submit flow run 120 sec earlier than scheduled, no changes to how Prefect requests pods
v
Can you explain the difference between flow run submission and pod request?
a
agent submits flow runs to infra, infra block requests pods - you can check the code for K8s block, the run method may make it more clear
the effect should be: flow run should start 120 sec earlier than it would otherwise
v
What is the benefit of this feature? We expected pod will be created and wait for scheduled start time to run flow code as documentation says.
a
flow run starts 120 sec earlier than scheduled - that's the benefit
v
But resources were requested at scheduled start time, not 120 seconds before.
a
please open an issue with more detailed description, I don't understand the problem here
scheduled time is set based on a deployment schedule, it has nothing to do with infra so I'm confused what you mean
v
Copy code
Imagine that you want to schedule a specific flow to run at 9 AM, but it roughly takes a minute to provision a serverless container. With this new CLI flag, your Prefect agent will begin flow run submission earlier than scheduled (in the example below, 60 seconds earlier) to ensure your container starts on time:
From this this example I understand that serverless container creation should start 60 seconds before schedule time (9 AM). Correct?
a
yes
v
Then why we can't expect that pod should be requested 120 seconds before schedule?
z
It's possible the transition to the pending state is blocked until the scheduled time, which would make prefetch not work as intended.
If so, it's a mistake in the prefetch concept and we will investigate a fix.
v
Should I create a bug?
z
Yes please! I'll look into it today.
v
Thanks, will post link here
3 Views
Open in Slack
PreviousNext
Powered by