Vadym Dytyniak
11/18/2022, 1:53 PMprefetch-seconds
. It doesn't work like we expected. Details in the thread.prefect agent start --prefetch-seconds 120 -q k8s
Anna Geller
11/18/2022, 1:55 PMVadym Dytyniak
11/18/2022, 1:56 PM13:46:00
and prefetch-seconds=120
.
In the logs we see that agent started flow run submission, but pod needed for this flow run was not requested, it requests resources after 120 seconds.
Logs:
13:44:00.562 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:05.655 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:10.753 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:15.832 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:20.901 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:26.064 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:31.134 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:36.195 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:41.290 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:46.383 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:51.482 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:44:56.598 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:01.773 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:06.846 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:11.919 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:16.985 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:22.061 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:27.168 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:32.251 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:37.314 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:42.463 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:47.556 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:52.641 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:45:57.752 | INFO | prefect.agent - Submitting flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:46:02.420 | INFO | prefect.agent - Completed submission of flow run '5200da86-8b3d-4873-8375-88099f84c95d'
13:46:02.464 | INFO | prefect.infrastructure.kubernetes-job - Job 'theta5-klendeth-k-bkl22': Pod has status 'Pending'.
13:47:02.459 | ERROR | prefect.infrastructure.kubernetes-job - Job 'theta5-klendeth-k-bkl22': Pod never started.
Anna Geller
11/18/2022, 1:58 PMagent started flow run submission, but pod needed for this flow run was not requestedthis is only about submission, prefetch means we submit flow run 120 sec earlier than scheduled, no changes to how Prefect requests pods
Vadym Dytyniak
11/18/2022, 2:00 PMAnna Geller
11/18/2022, 2:01 PMVadym Dytyniak
11/18/2022, 2:07 PMAnna Geller
11/18/2022, 2:17 PMVadym Dytyniak
11/18/2022, 2:19 PMAnna Geller
11/18/2022, 2:20 PMVadym Dytyniak
11/18/2022, 2:25 PMImagine that you want to schedule a specific flow to run at 9 AM, but it roughly takes a minute to provision a serverless container. With this new CLI flag, your Prefect agent will begin flow run submission earlier than scheduled (in the example below, 60 seconds earlier) to ensure your container starts on time:
From this this example I understand that serverless container creation should start 60 seconds before schedule time (9 AM). Correct?Anna Geller
11/18/2022, 2:28 PMVadym Dytyniak
11/18/2022, 2:29 PMZanie
11/18/2022, 3:11 PMVadym Dytyniak
11/18/2022, 3:12 PMZanie
11/18/2022, 3:13 PMVadym Dytyniak
11/18/2022, 3:13 PM