Hi We are trying new agent flag released in Prefect 2 6 8 `p

Question

Hi We are trying new agent flag released in Prefect 2 6 8 `prefetch seconds` It doesn t work like we expected Details in the thread

Vadym Dytyniak · Answer

Agent start command `prefect agent start prefetch seconds 120 q k8s`

Anna Geller · Answer

before you go on describing here GitHub issue looks like a bug

Vadym Dytyniak · Answer

Flow was scheduled to run at `13 46 00` and `prefetch seconds=120` In the logs we see that agent started flow run submission but pod needed for this flow run was not requested it requests resources after 120 seconds Logs ```13 44 00 562 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 05 655 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 10 753 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 15 832 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 20 901 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 26 064 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 31 134 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 36 195 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 41 290 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 46 383 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 51 482 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 44 56 598 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 01 773 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 06 846 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 11 919 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 16 985 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 22 061 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 27 168 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 32 251 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 37 314 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 42 463 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 47 556 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 52 641 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 45 57 752 | INFO | prefect agent Submitting flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 46 02 420 | INFO | prefect agent Completed submission of flow run 5200da86 8b3d 4873 8375 88099f84c95d 13 46 02 464 | INFO | prefect infrastructure kubernetes job Job theta5 klendeth k bkl22 Pod has status Pending 13 47 02 459 | ERROR | prefect infrastructure kubernetes job Job theta5 klendeth k bkl22 Pod never started ```

Vadym Dytyniak · Answer

< Anna Geller> I will create a bug but currently I am not sure it is expected behaviour or not Maybe I misunderstood the feature description Please clarify

Anna Geller · Answer

ahh you re right it may not be a bug actually

Anna Geller · Answer

gt agent started flow run submission but pod needed for this flow run was not requested this is only about submission prefetch means we submit flow run 120 sec earlier than scheduled no changes to how Prefect requests pods

Vadym Dytyniak · Answer

Can you explain the difference between flow run submission and pod request

Anna Geller · Answer

agent submits flow runs to infra infra block requests pods you can check the code for K8s block the run method may make it more clear

Anna Geller · Answer

the effect should be flow run should start 120 sec earlier than it would otherwise

Vadym Dytyniak · Answer

What is the benefit of this feature We expected pod will be created and wait for scheduled start time to run flow code as <https deploy preview 7498 prefect orion netlify app concepts work queues configuring prefetch|documentation> says

Anna Geller · Answer

flow run starts 120 sec earlier than scheduled that s the benefit

Vadym Dytyniak · Answer

But resources were requested at scheduled start time not 120 seconds before

Anna Geller · Answer

please open an issue with more detailed description I don t understand the problem here

Anna Geller · Answer

scheduled time is set based on a deployment schedule it has nothing to do with infra so I m confused what you mean

Vadym Dytyniak · Answer

```Imagine that you want to schedule a specific flow to run at 9 AM but it roughly takes a minute to provision a serverless container With this new CLI flag your Prefect agent will begin flow run submission earlier than scheduled in the example below 60 seconds earlier to ensure your container starts on time ``` From this this example I understand that serverless container creation should start 60 seconds before schedule time 9 AM Correct

Anna Geller · Answer

yes

Vadym Dytyniak · Answer

Then why we can t expect that pod should be requested 120 seconds before schedule

Zanie · Answer

It s possible the transition to the pending state is blocked until the scheduled time which would make prefetch not work as intended

Zanie · Answer

If so it s a mistake in the prefetch concept and we will investigate a fix

Vadym Dytyniak · Answer

Should I create a bug

Zanie · Answer

Yes please I ll look into it today

Vadym Dytyniak · Answer

Thanks will post link here

Vadym Dytyniak · Answer