< Marvin> Why does my EKS Worker take 23 minutes to start a Prefect Community #ask-marvin

<@ULVA73B9P> Why does my EKS Worker take 23 minute...

Dave D

10/08/2024, 10:37 AM

@Marvin Why does my EKS Worker take 23 minutes to start a pod? When using a prefect agent it took only 2-3 minutes. There seems to be a 19 minutes and 15 seconds delay between "Opening process..." step and the start of the flow "Starting my_flow_hello_world

Copy code

Worker 'KubernetesWorker aaa-bbb-ccc-dddd-eeee' submitting flow run '1111-2222-3333-444-55555'
05:55:56 AM
prefect.flow_runs.worker
Creating Kubernetes job...
05:55:56 AM
prefect.flow_runs.worker
Job 'the_pod_name': Starting watch for pod start...
05:55:56 AM
prefect.flow_runs.worker
Job 'the_pod_name': Pod 'the_pod_name-9a9a9' has started.
05:55:56 AM
prefect.flow_runs.worker
Job 'the_pod_name': Pod has status 'Pending'.
05:55:56 AM
prefect.flow_runs.worker
Completed submission of flow run '1111-2222-3333-444-55555'
05:55:57 AM
prefect.flow_runs.worker
Job 'the_pod_name': Pod 'the_pod_name-9a9a9' has started.
05:55:57 AM
prefect.flow_runs.worker
Job 'the_pod_name': Pod 'the_pod_name-9a9a9' has started.
05:57:55 AM
prefect.flow_runs.worker
Job 'the_pod_name': Pod has status 'Running'.
05:57:55 AM
prefect.flow_runs.worker
Opening process...
05:58:02 AM
prefect.flow_runs.runner
Starting my_flow_hello_world
06:17:17 AM
prefect.flow_runs

Marvin

10/08/2024, 11:03 AM

Your message was too long by 154 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

Why does my EKS Worker take 23 minutes to start a pod? When using a prefect agent it took only 2-3 minutes. There seems to be a 19 minutes and 15 seconds delay between "Opening process..." step and the start of the flow "Starting my_flow_hello_world

```Worker 'KubernetesWorker aaa-bbb-ccc-dddd-eeee' submitting flow run '1111-2222-3333-444-55555' 055556 AM prefect.flow_runs.worker Creating Kubernetes job... 055556 AM prefect.flow_runs.worker Job 'the_pod_name': Starting watch for pod start... 055556 AM prefect.flow_runs.worker Job 'the_pod_name': Pod 'the_pod_name-9a9a9' has started. 055556 AM prefect.flow_runs.worker Job 'the_pod_name': Pod has status 'Pending'. 055556 AM prefect.flow_runs.worker Completed submission of flow run '1111-2222-3333-444-55555' 055557 AM prefect.flow_runs.worker Job 'the_pod_name': Pod 'the_pod_name-9a9a9' has started. 055557 AM prefect.flow_runs.worker Job 'the_pod_name': Pod 'the_pod_name-9

Dave D

10/08/2024, 11:05 AM

@Marvin Why does my EKS Worker take 23 minutes to start a pod? There seems to be a 19 minutes and 15 seconds delay between "Opening process..." step and the start of the flow "Starting my_flow_hello_world Worker 'KubernetesWorker kwid' submitting flow run 'flowid' 055556 AM prefect.flow_runs.worker Creating Kubernetes job... 055556 AM prefect.flow_runs.worker Job 'thepod': Starting watch for pod start... 055556 AM prefect.flow_runs.worker Job 'thepod': Pod 'thepod-a' has started. 055556 AM prefect.flow_runs.worker Job 'thepod': Pod has status 'Pending'. 055556 AM prefect.flow_runs.worker Completed submission of flow run 'flowid' 055557 AM prefect.flow_runs.worker Job 'thepod': Pod 'thepod-a' has started. 055557 AM prefect.flow_runs.worker Job 'thepod': Pod 'thepod-a' has started. 055755 AM prefect.flow_runs.worker Job 'thepod': Pod has status 'Running'. 055755 AM prefect.flow_runs.worker Opening process... 055802 AM prefect.flow_runs.runner Starting my_flow_hello_world 061717 AM

Copy code

prefect.flow_runs

Marvin

10/08/2024, 11:05 AM

Your message was too long by 82 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

Why does my EKS Worker take 23 minutes to start a pod? When using a prefect agent it took only 2-3 minutes. There seems to be a 19 minutes and 15 seconds delay between "Opening process..." step and the start of the flow "Starting my_flow_hello_world

Dave D

10/08/2024, 11:07 AM

Copy code

prefect.flow_runs

Open in Slack

Previous Next