https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> what is the reason for a worker going...
# ask-marvin
s
@Marvin what is the reason for a worker going offline and new a worker starting, is this handled by prefect cloud?
m
✅ thought for 144.6 seconds
Short answer: no — Prefect Cloud does not automatically restart or replace workers. What usually happens is: - your worker process/container/pod stops heartbeating - Prefect Cloud marks that worker offline - your own infrastructure may then restart it or start a replacement worker So if you see “worker went offline” and “a new worker started,” that is typically because of your runtime environment, such as: - Kubernetes restarting a pod - Docker restart policy - ECS/Fargate replacing a task - systemd/supervisor restarting a process - a VM/container redeploy How Prefect decides a worker is offline - Workers heartbeat to the API regularly - Default heartbeat interval is about 30 seconds - If Prefect doesn’t receive heartbeats for about 3 intervals (~90 seconds by default), the worker is shown as offline Common reasons a worker goes offline - worker process crashed - container/pod was restarted - node/VM restarted - temporary network loss to Prefect API/Cloud - auth/config issue (
PREFECT_API_URL
, 
PREFECT_API_KEY
) - the worker got stuck and stopped communicating Important distinction - Prefect Cloud: tracks worker status - Your infrastructure: starts/stops/restarts workers So the “new worker” is not generally created by Prefect Cloud itself; it’s usually created by the environment where you’re running the worker. If you want, I can help you narrow this down based on where the worker is running (Kubernetes, Docker, ECS, VM, etc.) and suggest the most likely reason.
s
@Marvin the worker is running in EKS
m
✅ thought for 8.1 seconds
In EKS, the most likely explanation is: - the worker pod stopped or was replaced by Kubernetes - Prefect Cloud noticed heartbeats stopped and marked the old worker offline - the replacement pod started a new worker process, which registered as a new worker So again, Prefect Cloud is not restarting it — Kubernetes in EKS is. Common EKS-specific reasons - pod eviction due to node pressure - node rotation / node autoscaling / managed node group update - deployment rollout creating a new pod - liveness/readiness probe failure causing restart - OOMKilled / crash loop - network interruption preventing heartbeats to Prefect Cloud - pod rescheduled to another node What to check in EKS 1. Check pod status/history - 
kubectl get pods -n <namespace>
- 
kubectl describe pod <worker-pod> -n <namespace>
2. Check for restarts / termination reason - look for: - 
OOMKilled
- 
Evicted
- 
CrashLoopBackOff
- probe failures - node shutdown/drain messages 3. Check deployment/rollout history - 
kubectl rollout history deployment/<deployment-name> -n <namespace>
4. Check worker logs - 
kubectl logs <worker-pod> -n <namespace> --previous
- and current logs too: - 
kubectl logs <worker-pod> -n <namespace>
5. Check node events - 
kubectl get events -n <namespace> --sort-by=.lastTimestamp
If the worker is managed by a 
Deployment
, a replaced pod will normally create what looks like a “new worker” in Prefect because the worker process re-registers. A useful mental model is: - Prefect worker identity is tied to the running process instance - when the pod dies and a new pod comes up, Prefect often sees that as a new worker instance Most likely culprit in EKS If this is happening unexpectedly, the top things I’d investigate first are: - liveness probe too aggressive - memory limit too low leading to 
OOMKilled
- node autoscaling or node replacement - rolling deployment / Helm upgrade If you want, paste: - your worker manifest/Helm values - 
kubectl describe pod ...
- the previous pod logs and I can help pinpoint exactly why it’s cycling.
s
@Marvin we have noticed the container is OOMKilled, are there any Prefect GitHub issues for known worker memory leaks in 
2.16.6
?
m
✅ thought for 19.9 seconds
I didn’t find anything specifically tied to 
2.16.6
 by exact version search, but I did find a very relevant known issue for the Kubernetes worker in the 2.16.x line: - 
KubernetesWorker
memory usage increases over time — #12989 - Reported against Prefect 2.16.0 - Describes worker memory increasing over time in Kubernetes - The report specifically mentions the worker container, not just flow-run pods So while I can’t confirm “2.16.6 has a confirmed worker memory leak” from GitHub issues alone, there is a known closely related issue in the same minor series. A few important notes: - I did not find an issue explicitly saying “memory leak fixed in 2.16.6” or “2.16.6 worker OOM leak” - The strongest match is 2.16.0 Kubernetes worker memory growth: #12989 I also found some related-but-less-direct issues: - OOM/state handling issues in Kubernetes, but mostly in 3.x, not your version - Other memory issues involving Dask/server/components, not necessarily worker leaks My practical read: If your EKS worker on 
2.16.6
is being 
OOMKilled
, issue #12989 is the closest known upstream signal that this may not be unique to your setup. What I’d recommend 1. Upgrade if possible - 
2.16.6
is pretty old now, and if you can move forward, that’s the best way to rule out known worker issues. - If feasible, I’d strongly consider moving toward Prefect 3.x rather than spending too much time hardening 2.16.6 behavior. 2. Increase worker memory temporarily - As a mitigation, raise the worker pod memory limit/request so you can observe whether usage steadily climbs over time. 3. Capture evidence - graph worker RSS over time - note whether growth correlates with: - number of flow runs polled - number of submitted k8s jobs - logs/events retention - failed/crashed runs 4. Check whether this is worker memory vs flow-run pod memory - in your case, it sounds like the worker container is OOMKilled, not the spawned job pod - that distinction matters a lot If you want, I can also help you with a targeted triage plan for EKS on 2.16.6 to determine whether this looks like: - known worker memory growth - too-low memory limits - excessive polling/submission load - Kubernetes API client buildup - log buffering / event accumulation
s
@Marvin I have a Prefect v3 worker up and running. I would like to deployer a basic flow to run on the worker. It would be good if I can deploy it from the command line for testing purposes. Can you help?
m
✅ thought for 92.2 seconds
Absolutely — if you already have a Prefect 3 worker running, the simplest way to test it from the command line is to: 1. create a tiny flow file 2. deploy it with 
prefect deploy
3. trigger it with 
prefect deployment run
Here’s a minimal example. 1. Create a flow file Save this as `hello_flow.py`:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow()
2. Deploy it to your existing work pool Assuming your worker is already listening on a work pool named something like `my-pool`:
Copy code
bash
prefect deploy hello_flow.py:hello_flow --name test-deployment --pool my-pool
That creates a deployment for the flow. 3. Trigger a run from the CLI
Copy code
bash
prefect deployment run hello-flow/test-deployment
If you want to pass a parameter:
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
A few notes: - 
hello_flow
becomes the flow name 
hello-flow
- 
test-deployment
is the deployment name you chose - the deployment path is usually 
<flow-name>/<deployment-name>
4. Watch it in the UI You should then see the run picked up by your worker in Prefect Cloud / Server UI. If you are not sure what your pool name is You can list work pools with:
Copy code
bash
prefect work-pool ls
If you want a one-command test setup If you don’t already have a pool/worker, the usual flow is:
Copy code
bash
prefect work-pool create --type process my-pool
prefect worker start --pool my-pool
prefect deploy hello_flow.py:hello_flow --name test-deployment --pool my-pool
prefect deployment run hello-flow/test-deployment
Important for remote workers If your worker is running somewhere else (like EKS, Docker, etc.), the worker must be able to access the flow code. For simple local testing, this works best when: - using a 
process
worker locally, or - the flow code is baked into the image / available to the worker environment If your worker is in EKS, I can give you the correct deployment pattern for Kubernetes workers, because that usually means you need either: - a Docker image containing the flow code, or - 
flow.from_source(...).deploy(...)
pointing at a git repo If you want, tell me: - your work pool type (
process
, 
docker
, 
kubernetes
, etc.) - whether your worker is local or in EKS and I’ll give you the exact commands for that setup.
s
@Marvin work pool type is kubernetes and worker is in EKS
m
✅ thought for 80.7 seconds
Yes — with a Kubernetes work pool and a worker in EKS, the key thing is: your worker cannot access a local 
.py
file on your laptop unless the flow code is packaged or pulled remotely. So for command-line testing, the two practical options are: 1. build/push a Docker image containing the flow code 2. deploy from a git repo that the run can pull from For EKS, the simplest and most reliable test path is usually Docker image + 
prefect deploy
. --- Option A: easiest for EKS — use a Docker image 1. Create a simple flow Save `hello_flow.py`:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")
--- 2. Create a 
prefect.yaml
 In the same directory, create:
Copy code
yaml
name: hello-project

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.6.0
      image_name: YOUR_ECR_IMAGE
      tag: latest
      dockerfile: auto
      platform: linux/amd64

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.6.0
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/flows

deployments:
  - name: test-deployment
    entrypoint: hello_flow.py:hello_flow
    work_pool:
      name: YOUR_K8S_POOL
      job_variables:
        image: "{{ build-image.image }}"
Replace: - 
YOUR_ECR_IMAGE
with something like 
<http://123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-test|123456789012.dkr.ecr.us-east-1.amazonaws.com/prefect-test>
- 
YOUR_K8S_POOL
with your Kubernetes work pool name --- 3. Log into ECR
Copy code
bash
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <http://123456789012.dkr.ecr.us-east-1.amazonaws.com|123456789012.dkr.ecr.us-east-1.amazonaws.com>
If the repo does not exist yet, create it:
Copy code
bash
aws ecr create-repository --repository-name prefect-test
--- 4. Deploy from the command line
Copy code
bash
prefect deploy
That should: - build the image - push it to ECR - create the deployment in Prefect --- 5. Trigger a test run
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
Then your EKS worker should pick it up and create a job pod. --- Option B: deploy from git instead of building an image If your code is already in GitHub, you can avoid a Docker build and use source-based deployment. Example:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    flow.from_source(
        source="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Run:
Copy code
bash
python deploy.py
This works well if: - the repo is public, or - your runtime has credentials to pull it --- Important note for EKS If you use a Kubernetes work pool, the actual run happens in a new Kubernetes job/pod. That pod needs: - an image it can pull, or - a repo it can clone - any required Python dependencies installed So a plain command like:
Copy code
bash
prefect deploy hello_flow.py:hello_flow --name test-deployment --pool my-pool
is not usually sufficient by itself for EKS unless the runtime already knows how to get your code. --- Recommended test path for you Since you want a basic command-line test, I’d use: - 
hello_flow.py
- 
prefect.yaml
- ECR-backed image - 
prefect deploy
- 
prefect deployment run
That’s the most straightforward path for Kubernetes/EKS. Useful docs - Deployments in Prefect 3 - Work pools - Workers - Kubernetes deployment patterns
If you want, I can give you a copy-paste-ready example for: - ECR + 
prefect.yaml
, or - GitHub repo + 
flow.from_source(...).deploy(...)
 using your actual: - AWS region - ECR repo name - Kubernetes work pool name
s
@Marvin I don't have access to ECR, what are the other alternatives?
m
✅ thought for 15.2 seconds
Yes — you still have good options without ECR. For an EKS Kubernetes work pool, your main alternatives are: 1. Use a public or private Docker registry other than ECR Examples: - Docker Hub - GHCR - GitLab Container Registry - Quay If your EKS cluster can pull from that registry, this works just like ECR. 2. Deploy from a git repository This is often the best no-ECR option for testing. With Prefect 3.x, you can deploy from source hosted in GitHub/GitLab instead of packaging the code into an image. Example:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Then run:
Copy code
bash
python deploy.py
After that:
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
Important caveat: The execution environment still needs a Python environment with your dependencies. For a very basic flow, the default Prefect image may be enough. If your flow needs extra packages, then you’ll eventually need a custom image somewhere. 3. Use a public repo for a very simple test If your flow is tiny and only uses Prefect itself, you can: - put the flow in a public GitHub repo - deploy from that repo - let the Kubernetes job pull the code at runtime That is usually the fastest test path if you don’t control container registries. 4. Ask whether your Kubernetes work pool already has a usable base image Sometimes a team already has: - an internal registry - a prebuilt Prefect image - a work pool 
base_job_template
with a standard image If so, you may be able to use git-based source retrieval and run on that existing image without needing to publish your own image. --- What usually will not work These are not good fits for EKS workers: - local file only on your laptop - 
process
-style assumptions for code availability - deployments that assume the worker pod can see your local filesystem --- Best recommendation for your situation If you want the simplest test without ECR, I recommend: - create a small public GitHub repo - add 
hello_flow.py
- deploy with 
hello_flow.from_source(...).deploy(...)
- run it from CLI Example 
hello_flow.py
in GitHub repo:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")
Example 
deploy.py
on your machine:
Copy code
python
from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Then:
Copy code
bash
python deploy.py
prefect deployment run hello-flow/test-deployment --param name=marvin
Docs - Deployments - Work pools - Workers If you want, I can give you a complete GitHub-based example for: - public repo, or - private repo with credentials
s
@Marvin, if I use a GitHub repo to store the 
hello_flow.py
and use the following code to deploy it
Copy code
python
from prefect import flow

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Do I need a job manifest as well?
m
✅ thought for 13.1 seconds
Usually, no — you do not need to provide a separate Kubernetes job manifest just to make this work. If you deploy to a Kubernetes work pool, Prefect uses that work pool’s job template / base configuration to create the Kubernetes Job for each flow run. So with:
Copy code
python
hello_flow.from_source(
    source="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
    entrypoint="hello_flow.py:hello_flow",
).deploy(
    name="test-deployment",
    work_pool_name="YOUR_K8S_POOL",
)
Prefect will: - create the deployment - store the source location and entrypoint - when a run is triggered, the worker will generate the Kubernetes Job from the work pool configuration So the real question is: Does your Kubernetes work pool already have a usable base job template? In most setups, yes. If your EKS worker is already running and your Kubernetes work pool was created normally, there is already a default job template. You only need to customize the job template / manifest if you need things like: - custom namespace - service account - image pull secrets - CPU/memory requests and limits - node selectors / tolerations - custom environment variables - volumes / secrets / config maps For a basic hello world test, the default template is often enough. One important detail though Even if you don’t need a custom manifest, the job still needs an image to run in. That image must contain: - Python - Prefect - git support if it needs to clone a repo - any dependencies your flow needs So for a simple 
hello_flow.py
, this can work if the work pool’s default image is valid. If the default image in the work pool is a Prefect image, you may be fine for a basic test. When you would need to edit the work pool job template You’d likely need that if: - the image doesn’t have git installed - the pod needs GitHub credentials for a private repo - the pod needs more memory/CPU - the pod needs IAM/service account config - the namespace or secrets are wrong So: - No, you do not need to hand-write a Kubernetes job manifest for a simple test - Yes, you may need to adjust the work pool’s job template if the default runtime image/config is insufficient Practical recommendation Try it first with the existing work pool. If it fails, the most likely errors will be: - image/runtime problem - repo access problem - missing git - import/dependency problem You can inspect that from: - Prefect UI run logs - worker logs in EKS - spawned job/pod logs in Kubernetes If you want, I can help you verify whether your work pool’s current base job template is sufficient before you test.
s
@Marvin, our GitHub repo is setup so it is private and I'll need to apply a Github PAT, how will this change the code?
m
✅ thought for 91.4 seconds
Yes — for a private GitHub repo, you should switch from a plain URL string to a 
GitRepository(...)
object with credentials. The recommended pattern is to store the PAT in a Prefect 
Secret
block, then reference it in the deployment code. Here’s how your code would change. Recommended approach: use a Prefect Secret block
Copy code
python
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={
                "access_token": Secret.load("github-pat")
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
First, create the Secret block once You can do that from Python:
Copy code
python
from prefect.blocks.system import Secret

Secret(value="ghp_your_actual_token").save("github-pat", overwrite=True)
After that, your deployment code can load it with:
Copy code
python
Secret.load("github-pat")
What this does - Prefect stores the PAT securely as a block - the deployment stores a reference to that block - when the worker executes the deployment, Prefect can use that token to clone the private repo Less recommended: inline PAT You can also do this directly:
Copy code
python
from prefect import flow
from prefect.runner.storage import GitRepository

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={
                "access_token": "ghp_your_actual_token"
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
But I would not recommend this because the token is now living in code. For EKS specifically You still do not need to hand-write a Kubernetes job manifest just because the repo is private. You may, however, need to ensure: - the runtime image used by the Kubernetes work pool has 
git
available - the worker/job pod can reach GitHub over the network - the pod can reach your Prefect API/backend to resolve the 
Secret
block Minimal flow of steps 1. Save the PAT as a Prefect Secret block 2. Use 
GitRepository(...)
with 
credentials={"access_token": Secret.load("github-pat")}
3. Run the deploy script 4. Trigger with CLI Example trigger:
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
Docs - Deploy via Python - Work pools - Workers If you want, I can give you a full copy-paste example with: - 
save_secret.py
- 
deploy.py
- expected repo layout - a quick checklist for whether your Kubernetes work pool image can clone from GitHub
s
@Marvin could I use an environment variable instead of using a Prefect secret?
m
✅ thought for 11.9 seconds
Yes — you can use an environment variable instead of a Prefect Secret. For example:
Copy code
python
import os
from prefect import flow
from prefect.runner.storage import GitRepository

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={
                "access_token": os.environ["GITHUB_PAT"]
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Then set it before running the deploy script:
Copy code
bash
export GITHUB_PAT=ghp_your_token_here
python deploy.py
But there’s an important distinction: This environment variable is used when you create the deployment, and depending on how the deployment is stored/resolved, you may also need the credential available at runtime for the Kubernetes job that actually clones the repo. So in practice, for EKS, there are two places env vars might matter: 1. On your machine / CI when running 
deploy.py
2. In the Kubernetes job environment when the flow run starts If the token is only resolved during deployment creation, this may be enough. If the runtime needs to clone from GitHub using that credential, the job may also need access to it. Safer EKS pattern If you want to use env vars, I’d recommend passing the env var into the job environment too, via work pool job variables if needed. Conceptually:
Copy code
python
import os
from prefect import flow
from prefect.runner.storage import GitRepository

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    token = os.environ["GITHUB_PAT"]

    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={
                "access_token": token
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
        job_variables={
            "env": {
                "GITHUB_PAT": token
            }
        },
    )
That said, be careful: passing PATs around as plain env vars is less secure than using a Prefect Secret or Kubernetes Secret. Best-practice guidance - Okay for quick testing: environment variable - Better for ongoing use: Prefect Secret block or Kubernetes Secret - Best for EKS production setups: inject the PAT from a Kubernetes Secret into the job pod So the answer is: - Yes, you can use an environment variable - For a quick test, that’s fine - For private GitHub + EKS, make sure the runtime pod can also authenticate if needed If you want, I can show you: 1. a quick local-env-var test version, or 2. a more correct EKS version using a Kubernetes Secret
s
@Marvin is this the only code I need that contains both the flow and the deployment code? I push code to GitHub and then run the deployment locally?
Copy code
import os
from prefect import flow
from prefect.runner.storage import GitRepository

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={
                "access_token": os.environ["GITHUB_PAT"]
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
m
✅ thought for 15.8 seconds
Yes — for a basic test, that can be the only Python file you need. Your workflow would be: 1. save that file as 
hello_flow.py
2. push it to your private GitHub repo 3. set 
GITHUB_PAT
locally 4. run the file locally to create the deployment 5. trigger the deployment from the CLI or UI So conceptually: yes, one file can contain both the flow and the deployment code. That said, I’d make two small fixes to your example. Corrected version
Copy code
python
import os
from prefect import flow
from prefect.runner.storage import GitRepository

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={
                "access_token": os.environ["GITHUB_PAT"]
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Two important corrections: - remove the Slack-style angle brackets around the GitHub URL - use 
<https://github.com/>...
- not 
<https://github.com/...>
- make sure 
entrypoint="hello_flow.py:hello_flow"
matches the actual filename and flow function name in the repo How to use it 1. Put 
hello_flow.py
in your repo 2. Commit and push it 3. Locally, export your PAT:
Copy code
bash
export GITHUB_PAT=ghp_your_token_here
4. Run the deployment script locally:
Copy code
bash
python hello_flow.py
That creates the Prefect deployment. 5. Trigger it:
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
--- What this file is doing - defines the flow: 
hello_flow
- when run as a script, tells Prefect: - where to fetch the code from - how to find the flow in that repo - which work pool to deploy to That means the same file can act as: - flow source code in GitHub - deployment creation script when run locally --- One subtle caveat for private repos This approach often works well, but you should know the PAT is being read locally during deployment creation. At runtime, the Kubernetes job may also need to authenticate to GitHub to pull the repo. If the deployment stores the repo credentials with the source configuration, you may be okay. If not, you may need to make credentials available to the runtime too. For a simple first test, I’d try this exact pattern first. --- What I would verify if it fails If the deployment is created but the run fails, check: - worker logs - Kubernetes job/pod logs - whether the runtime image has 
git
- whether the job pod can reach GitHub - whether the repo credential is available at runtime --- Very small improvement If you want to make it a bit cleaner, you can separate flow definition and deployment logic:
Copy code
python
import os
from prefect import flow
from prefect.runner.storage import GitRepository

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

def deploy():
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            credentials={"access_token": os.environ["GITHUB_PAT"]},
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )

if __name__ == "__main__":
    deploy()
Not required, just a little tidier. So the short answer is: yes, that single file is enough for a basic test. If you want, I can also give you a full end-to-end checklist for this exact private-GitHub-to-EKS test, including what success/failure should look like in Prefect and Kubernetes.
s
@Marvin do I need to commit and push this code to my GitHub repo and when there I can then run python deploy?
m
✅ thought for 6.2 seconds
Yes — you should commit and push it first. Because your deployment points to:
Copy code
python
source=GitRepository(
    url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
    ...
)
the worker in EKS will pull the code from GitHub, not from your local machine. So the file needs to exist in the repo before you create and run the deployment. Recommended order 1. Create 
hello_flow.py
2. Commit it 3. Push it to GitHub 4. Export your PAT locally 5. Run the script locally to create the deployment Example:
Copy code
bash
git add hello_flow.py
git commit -m "Add hello flow"
git push
export GITHUB_PAT=ghp_your_token_here
python hello_flow.py
Then trigger the deployment:
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
Why this order matters If you run 
python hello_flow.py
before the code is in GitHub, the deployment may still get created, but when the worker tries to execute it, it may fail because: - the file is not in the repo yet - the entrypoint doesn’t exist in the pushed branch - the worker pulls an older version of the repo So yes: push first, then run the deployment script locally. One more thing to double-check Make sure the deployment is referencing the correct branch if needed. If your code is not on the repo’s default branch, you may want to specify it in 
GitRepository(...)
. Example:
Copy code
python
GitRepository(
    url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
    branch="main",
    credentials={"access_token": os.environ["GITHUB_PAT"]},
)
That can help avoid confusion. So the shortest answer is: - Yes, commit and push first - Then run the deployment script locally
s
@Marvin using this code
Copy code
GitRepository(
    url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
    branch="main",
    credentials={"access_token": os.environ["GITHUB_PAT"]},
)
I receive the error
Copy code
ValueError: Please save your access token as a Secret block before converting this storage object to a pull step.
m
✅ thought for 12.3 seconds
That error is expected with this deployment path. When Prefect converts 
GitRepository(...)
into the deployment’s pull step, it does not allow a raw access token string there. It expects the token to come from a saved Prefect Secret block. So for 
flow.from_source(...).deploy(...)
, a plain env var passed directly as 
access_token
is not sufficient in this case. What to do instead 1. Save the PAT as a Prefect Secret block 2. Reference that block in 
GitRepository
Example:
Copy code
python
from prefect.blocks.system import Secret

Secret(value="ghp_your_token").save("github-pat", overwrite=True)
Then update your deployment code to:
Copy code
python
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            branch="main",
            credentials={
                "access_token": Secret.load("github-pat")
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Important note Also remove the angle brackets from the URL. Use:
Copy code
python
url="<https://github.com/YOUR_ORG/YOUR_REPO.git>"
not:
Copy code
python
url="<https://github.com/YOUR_ORG/YOUR_REPO.git>"
Why this happens Prefect needs to serialize the storage config for the deployment. For security reasons, when converting to a pull step it requires a Secret block reference instead of embedding a raw token value. So in this specific workflow: - env var only → not accepted for the Git pull step conversion - Prefect Secret block → accepted Minimal setup sequence 1. Save secret once:
Copy code
python
from prefect.blocks.system import Secret
Secret(value="ghp_your_token").save("github-pat", overwrite=True)
2. Push your flow file to GitHub 3. Run the deployment script locally 4. Trigger the deployment run If you want to keep the token out of code entirely You can still use an env var to create the Secret block:
Copy code
python
import os
from prefect.blocks.system import Secret

Secret(value=os.environ["GITHUB_PAT"]).save("github-pat", overwrite=True)
Then use 
Secret.load("github-pat")
in the deploy script. So you still get env-var-based local setup, but Prefect stores it in the format required for deployment. If you want, I can give you the exact two-file setup: - 
save_secret.py
- 
hello_flow.py
with the precise order of commands to run.
s
@Marvin yes
m
✅ thought for 9.1 seconds
Absolutely — here’s the exact minimal setup. You’ll use two files: - 
save_secret.py
— run once (or whenever the PAT changes) - 
hello_flow.py
— contains both the flow and deployment code --- 1. 
save_secret.py
Copy code
python
import os
from prefect.blocks.system import Secret

if __name__ == "__main__":
    Secret(value=os.environ["GITHUB_PAT"]).save(
        "github-pat",
        overwrite=True,
    )
    print("Saved Prefect Secret block: github-pat")
This reads your local 
GITHUB_PAT
env var and stores it as a Prefect Secret block named 
github-pat
. --- 2. 
hello_flow.py
Copy code
python
from prefect import flow
from prefect.runner.storage import GitRepository
from prefect.blocks.system import Secret

@flow(log_prints=True)
def hello_flow(name: str = "world"):
    print(f"Hello, {name}!")

if __name__ == "__main__":
    hello_flow.from_source(
        source=GitRepository(
            url="<https://github.com/YOUR_ORG/YOUR_REPO.git>",
            branch="main",
            credentials={
                "access_token": Secret.load("github-pat")
            },
        ),
        entrypoint="hello_flow.py:hello_flow",
    ).deploy(
        name="test-deployment",
        work_pool_name="YOUR_K8S_POOL",
    )
Replace: - 
YOUR_ORG
- 
YOUR_REPO
- 
YOUR_K8S_POOL
--- 3. Push 
hello_flow.py
to GitHub first From your local repo:
Copy code
bash
git add hello_flow.py
git commit -m "Add hello flow"
git push
You do not need to push 
save_secret.py
unless you want to keep it in the repo for convenience. --- 4. Export your GitHub PAT locally
Copy code
bash
export GITHUB_PAT=ghp_your_actual_token
--- 5. Save the Prefect Secret block
Copy code
bash
python save_secret.py
--- 6. Create the deployment
Copy code
bash
python hello_flow.py
This creates the deployment in Prefect. --- 7. Run the deployment
Copy code
bash
prefect deployment run hello-flow/test-deployment --param name=marvin
--- What should happen - Prefect deployment gets created - your EKS worker picks up the run from the Kubernetes work pool - a Kubernetes job/pod gets created - the pod pulls the code from the private GitHub repo using the stored secret - the flow runs and prints 
Hello, marvin!
--- Important notes - The GitHub URL should be plain text:
Copy code
python
  "<https://github.com/YOUR_ORG/YOUR_REPO.git>"
not Slack-formatted with angle brackets - 
entrypoint="hello_flow.py:hello_flow"
must exactly match: - the file path in the repo - the flow function name If the file is in a subdirectory, use that path instead, for example:
Copy code
python
entrypoint="flows/hello_flow.py:hello_flow"
--- If it fails The main things to check are: - worker logs in EKS - job/pod logs for the spawned flow run - whether the repo path/branch is correct - whether the runtime image can clone GitHub - whether the job can reach Prefect API and GitHub --- Relevant docs - Deploy via Python - Workers - Work pools If you want, I can also give you: - a private repo in subfolder example, or - a troubleshooting checklist for the first EKS run
s
@Marvin how do I upgrade the worker as I am seeing the following message displayed?
Copy code
This worker's Prefect version is too low to display logs. Upgrade to Prefect version 3.1.1 or higher to view logs.
The worker should be running Prefect 3.7.5 but it still seems to be on 2.0.0
m
✅ thought for 14.4 seconds
That message usually means the running worker pod is still using an old Prefect image, even if you expected it to be 
3.7.5
. So the fix is not in Prefect Cloud — it’s in EKS/Kubernetes: you need to update the worker deployment/image and restart the pod(s). What’s likely happening - your work pool/worker was originally created with an older image - the EKS deployment is still running a container with 
prefect==2.0.0
- Prefect Cloud reports the version from the actual connected worker process So first, confirm what image/version the worker pod is really running. 1. Check the running worker pod image
Copy code
bash
kubectl get pods -n <namespace>
kubectl describe pod <worker-pod-name> -n <namespace>
Look for the 
Image:
line. You can also check directly:
Copy code
bash
kubectl get pod <worker-pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].image}'
2. Check the Prefect version inside the worker pod
Copy code
bash
kubectl exec -it <worker-pod-name> -n <namespace> -- prefect version
If that shows 
2.0.0
, then the worker really is old. --- How to upgrade it If you installed the worker with the Prefect Helm chart, the usual path is: 1. update the Helm repo 2. update your values if needed 3. run 
helm upgrade
4. verify the new pod comes up with the correct image Update Helm repo
Copy code
bash
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
Check current Helm release
Copy code
bash
helm list -n <namespace>
Then inspect current values:
Copy code
bash
helm get values <release-name> -n <namespace>
Upgrade the worker If your chart values specify an image, make sure it points to a 3.x image. Then run:
Copy code
bash
helm upgrade <release-name> prefect/prefect-worker -n <namespace> -f values.yaml
If you want to force the image explicitly, that depends on how your chart values are structured, but conceptually you want the worker image to be something like a 
prefecthq/prefect:3.7.5-python...
image, not a 
2.x
image. After upgrade:
Copy code
bash
kubectl rollout status deployment/<worker-deployment-name> -n <namespace>
--- Quick workaround: restart the deployment If you already updated the image but the old pod is still hanging around, restart it:
Copy code
bash
kubectl rollout restart deployment/<worker-deployment-name> -n <namespace>
Then verify the new pod version:
Copy code
bash
kubectl exec -it <new-worker-pod> -n <namespace> -- prefect version
--- What to look for in your Helm values Common causes: - pinned old image tag - old chart values preserved across upgrades - image override still points to a 2.0.0 image - multiple worker deployments exist, and the old one is still connected Check for: - 
image.repository
- 
image.tag
- worker-specific image overrides --- Also check if more than one worker is connected Sometimes the UI warning is from an older worker still online, not the one you think you upgraded. So check: - whether there are multiple workers in the same work pool - whether an old 
2.0.0
worker is still registered and heartbeating You may need to scale down/remove the old worker deployment. --- Practical checklist 1. 
kubectl exec ... -- prefect version
2. confirm actual container image 3. inspect Helm values for pinned old image 4. 
helm upgrade ...
5. 
kubectl rollout restart deployment/...
6. verify only the new worker is connected Docs - Workers - Kubernetes workers - Prefect Helm chart If you want, paste: - 
helm get values <release-name> -n <namespace>
- 
kubectl get deployment -n <namespace> -o yaml | grep -A5 -B5 image
- 
kubectl exec ... -- prefect version
and I can help pinpoint exactly why it’s still showing 
2.0.0
.
s
@Marvin this is the version that is running
Copy code
Version:              3.7.5
API version:          0.8.4
Python version:       3.12.13
Git commit:           756685ab
Built:                Thu, Jun 18, 2026 09:32 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          cloud
Pydantic version:     2.13.4
Server:
  Database:           sqlite
  SQLite version:     3.50.4
Integrations:
  prefect-kubernetes: 0.7.10
  prefect-redis:      0.2.12
m
✅ thought for 17.4 seconds
Thanks — that confirms the pod you checked is on Prefect 3.7.5. So if the UI still says:
Copy code
text
This worker's Prefect version is too low to display logs. Upgrade to Prefect version 3.1.1 or higher to view logs.
then the most likely explanation is: Prefect Cloud is associating that run or worker record with a different/older worker instance, not the pod you just checked. Most likely causes: - there is another worker still connected or recently connected with an old version - the worker record in Cloud is stale - the run was picked up by an older worker instance before the upgrade - you have multiple worker deployments/pods in the same work pool What to check next 1. Check for multiple worker pods in EKS
Copy code
bash
kubectl get pods -n <namespace> -o wide
Make sure there isn’t an older worker pod still running. --- 2. Check the image/version for every worker pod For each pod:
Copy code
bash
kubectl exec -it <pod-name> -n <namespace> -- prefect version
If any one of them reports 
2.0.0
, that’s probably your culprit. --- 3. Check the Prefect UI worker list Go to your work pool / workers page and see whether: - multiple workers are registered - one appears older/stale/offline - the run was assigned to a specific worker name that may not be the new one --- 4. Restart the worker deployment cleanly Even if the pod you checked is correct, I’d do a clean rollout restart:
Copy code
bash
kubectl rollout restart deployment/<worker-deployment-name> -n <namespace>
kubectl rollout status deployment/<worker-deployment-name> -n <namespace>
Then re-check the worker version in the new pod. --- 5. Verify the run is picked up by the restarted worker Trigger a new test deployment run after the restart. If old runs were started before the upgrade, the UI warning may still reflect the older worker metadata for those runs. Try:
Copy code
bash
prefect deployment run hello-flow/test-deployment
If new runs no longer show the warning, then it was tied to an older worker/run record. --- What I suspect most Since your live worker is definitely 
3.7.5
, I suspect one of these: - stale old worker record in Prefect Cloud - multiple workers in the pool - the specific run you’re viewing was started by an older worker before upgrade Important nuance That warning can be tied to the worker metadata attached to a run, not necessarily “what is running right now in Kubernetes.” So checking one current pod is necessary, but not always sufficient. Best next test 1. restart the worker deployment 2. confirm only one worker pod is running 3. launch a brand new flow run 4. see whether that new run still shows the warning If it does, then we may be looking at either: - a stale worker registration/version mismatch in Cloud - a bug in worker version reporting I wasn’t able to confirm an existing GitHub issue for that exact warning because GitHub issue search errored during lookup and no matching discussion turned up. If you want, I can help you narrow it down with just these outputs: - 
kubectl get pods -n <namespace>
- worker names shown in the Prefect UI - whether the warning appears on new runs after restart or only older runs
s
@Marvin the strange thing is I can see view the logs even though the Prefect Version 2.0.0 with the banner
m
✅ thought for 11.8 seconds
That is strange, but it actually suggests the banner is likely incorrect/stale UI metadata, not a real runtime limitation. Since: - your actual worker pod reports 3.7.5 - you can view logs the most likely conclusion is: the worker version shown in the UI is stale or misreported, while the actual worker process is new enough to support logs. So this is probably one of: - stale worker record/version metadata in Prefect Cloud - an old worker registration reused in the UI - a UI bug where the displayed version is wrong - a mismatch between the worker identity record and the actual connected worker process Why I think that If the worker were truly running 
2.0.0
, I would expect behavior to line up consistently: - old version shown - banner shown - logs unavailable But you have: - old version shown - banner shown - logs still available That points to the gating/banner being based on bad metadata, while the actual log retrieval path is functioning. Most likely root cause The worker name/record may already exist in Cloud from an older instance, and the version field shown in the UI may not have updated correctly when the newer worker connected. What I would do 1. Restart the worker deployment to force a fresh registration cycle
Copy code
bash
kubectl rollout restart deployment/<worker-deployment-name> -n <namespace>
2. Watch whether a new worker record appears in the UI - possibly with a different worker name/UUID suffix - see whether the displayed version changes to 
3.7.5
3. Check whether the old 
2.0.0
worker record eventually goes offline 4. Trigger a brand new run and see whether the banner still appears for that worker/run --- What this means operationally If logs are working, then this is probably not blocking execution. It looks more like a display/metadata issue than an actual worker upgrade problem. --- Could this be worth a bug report? Yes, potentially — especially if after restart/new runs: - the worker still displays 
2.0.0
- the actual pod is definitely 
3.7.5
- logs still work That would be strong evidence of a UI/backend worker-version reporting mismatch. I did try to search for an existing GitHub issue for this exact banner/version mismatch, but GitHub issue search errored at the time and I didn’t find an existing discussion. --- My short diagnosis - Your worker is almost certainly upgraded correctly - The UI version field/banner appears stale or wrong - Since logs work, this is likely cosmetic or metadata-related If you want, I can help you put together a minimal reproducible bug report with: - worker UI screenshot - 
prefect version
output from the pod - exact steps to reproduce so you can compare against known behavior or open an issue if needed.
s
@Marvin, I see 2 workers online now and the new one is still Prefect version 2.0.0
m
✅ thought for 11.3 seconds
That explains it: you still have an old worker online. So right now you likely have: - one worker process/pod running 3.7.5 - another worker process/pod still connected and reporting 2.0.0 That old one is what’s causing the banner/version confusion. What this means Prefect Cloud is not wrong — there really is still a 
2.0.0
worker connected somewhere. So the next step is to find where that second worker is running and shut it down. Most likely places the old worker could be coming from - another pod in the same EKS namespace - another deployment/statefulset in the cluster - an older Helm release still running - a worker running in another namespace - a local/manual worker process someone started - another environment pointed at the same Prefect workspace and work pool How to find it 1. Check all pods in the namespace
Copy code
bash
kubectl get pods -n <namespace> -o wide
2. Check all namespaces for Prefect worker pods
Copy code
bash
kubectl get pods --all-namespaces | grep -i prefect
3. Check deployments
Copy code
bash
kubectl get deployments --all-namespaces | grep -i prefect
4. Check Helm releases
Copy code
bash
helm list --all-namespaces
Look for: - multiple 
prefect-worker
releases - old agent/worker installs - multiple namespaces 5. Check the version inside each candidate pod For each worker-looking pod:
Copy code
bash
kubectl exec -it <pod-name> -n <namespace> -- prefect version
That should reveal which pod is still on 
2.0.0
. --- If you find the old worker You can stop it by: - scaling down the old deployment - uninstalling the old Helm release - deleting the old pod if it is managed elsewhere - stopping any manually run worker process Examples:
Copy code
bash
kubectl scale deployment <old-worker-deployment> --replicas=0 -n <namespace>
or
Copy code
bash
helm uninstall <old-release-name> -n <namespace>
--- Also check outside EKS If you don’t find it in your cluster, it may be: - running on someone’s laptop - running on another VM - running in another cluster/account - connected to the same Prefect workspace/work pool That’s especially possible if the worker was started manually with the same API key/workspace config. --- Why you saw logs anyway That makes more sense now too: - one worker is modern and supports logs - another old worker is still online and causing the version warning - UI may be surfacing the old worker version in places where it’s ambiguous which worker record is being referenced --- Recommended cleanup plan 1. identify both online workers in the UI by name 2. map those names to actual pods/processes 3. stop the old 
2.0.0
worker 4. leave only the 
3.7.5
worker online 5. trigger a fresh run and verify the banner disappears If you want, paste the output of:
Copy code
bash
kubectl get pods --all-namespaces | grep -i prefect
helm list --all-namespaces
and I can help you spot the likely old worker.
s
@Marvin is there any recommendation on the number of workers per work pool?
m
✅ thought for 113.9 seconds
Yes — multiple workers per work pool is a normal pattern. There isn’t a strict Prefect-recommended number like “always use 3,” but there are good rules of thumb. Short answer - 1 worker per work pool is fine for dev/test and simple setups - multiple workers per work pool is recommended for higher availability and more throughput - the right number depends more on your infrastructure and workload than on a hard Prefect limit How it works Multiple workers can poll the same work pool safely: - each worker asks Prefect for work - Prefect ensures a run is only claimed once - workers do not duplicate runs So having 2, 3, or more workers on one pool is supported. When 1 worker is enough Use a single worker if: - you’re testing - you have low run volume - you want simpler debugging - you don’t need high availability When multiple workers help Use multiple workers if: - you want redundancy - you want better throughput - you want to avoid a single worker outage stalling all work - you have bursts of many flow runs For EKS specifically, multiple worker pods are common. Important nuance for Kubernetes work pools For a Kubernetes work pool, the worker usually does not execute the flow itself. Instead, the worker: - polls for work - creates Kubernetes Jobs for flow runs So scaling workers mostly affects: - how much scheduling/submission capacity you have - resilience if one worker pod dies It does not directly mean “more flows run inside the worker pod.” The actual flow runs happen in separate Kubernetes job pods. Practical recommendation for EKS A common pattern is: - 1 worker for small/test setups - 2 workers for basic HA/redundancy - more than 2 only if you’re pushing meaningful scheduling volume or want extra resilience If you’re asking for a sensible default, I’d say: - Dev/test: 1 - Production small/medium: 2 - Larger scale: 2+ based on observed throughput and reliability needs Things to watch if you add workers - duplicate old worker installs, like the issue you just saw - Kubernetes resource usage for worker pods - API polling overhead - whether workers are actually the bottleneck, versus cluster capacity In many EKS setups, the bottleneck is more often: - cluster resources - image pull time - job startup latency - Git clone time - Kubernetes API rate/latency —not the number of Prefect workers. My recommendation for your case Given what you’ve already seen, I’d first aim for: - one clean 3.7.5 worker deployment - then move to two replicas only if you want redundancy Before scaling out, I’d make sure: - the old 2.0.0 worker is fully gone - worker version reporting is clean - a basic deployment runs successfully end-to-end Docs - Workers - Work pools - Kubernetes If you want, I can also suggest a Helm/EKS worker replica strategy for: - dev - staging - production
s
@Marvin what are the recommended resources for a worker that is used to trigger multiple prefect jobs?
m
✅ thought for 122.7 seconds
For a Kubernetes worker in EKS that mostly submits Prefect flow runs as Kubernetes jobs rather than running heavy user code itself, you can usually size it pretty modestly. Good starting point I’d recommend starting with:
Copy code
yaml
resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 1Gi
That’s a solid default for a worker whose main job is: - polling Prefect - heartbeating - creating Kubernetes Jobs - watching job/pod status If you trigger multiple jobs regularly If this worker is submitting a decent volume of jobs, I’d lean slightly more conservative on memory:
Copy code
yaml
resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 1Gi
That’s probably the best practical starting point if: - the worker is busy - you’ve already seen OOM behavior before - it is monitoring multiple concurrent runs When to go higher Consider:
Copy code
yaml
resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: 2000m
    memory: 2Gi
if: - one worker is managing lots of concurrent submissions - you have many pod/job state changes - you enable more Kubernetes observation features - you observe memory growth over time What drives worker resource usage Even though the worker is not running your actual flow logic, it still uses resources for: - polling the Prefect API - Kubernetes API calls - job creation and monitoring - caching pod/job state - log forwarding / crash observation - managing multiple concurrent flow-run submissions So in EKS, memory pressure often comes more from watching and coordinating many jobs than from CPU. My practical recommendation by environment Small / dev
Copy code
yaml
requests:
  cpu: 100m
  memory: 256Mi
limits:
  cpu: 500m
  memory: 512Mi
Typical production starting point
Copy code
yaml
requests:
  cpu: 250m
  memory: 512Mi
limits:
  cpu: 1000m
  memory: 1Gi
Busy production / many concurrent jobs
Copy code
yaml
requests:
  cpu: 500m
  memory: 1Gi
limits:
  cpu: 2000m
  memory: 2Gi
What I would do in your case Since you’ve already seen worker confusion and OOM history in the broader setup, I would start with:
Copy code
yaml
requests:
  cpu: 250m
  memory: 512Mi
limits:
  cpu: 1000m
  memory: 1Gi
Then monitor: - pod RSS / memory over time - restart count - OOMKilled events - worker responsiveness - job submission latency One more important lever: concurrency If one worker is triggering many jobs, also pay attention to the worker’s run limit. It’s often better to: - use reasonable worker concurrency - and/or 2 smaller worker replicas than one very overloaded worker Practical rule If the worker is only orchestrating jobs, memory is usually the first thing to increase, not CPU. Docs / references - Workers - Kubernetes workers - Prefect Helm chart If you want, I can turn this into a Helm values snippet for your EKS worker deployment.
s
@Marvin if I have a work pool that has two workers, are the workers load balanced so either can spin up a pod to run a job?
m
✅ thought for 11.0 seconds
Yes — effectively, yes. If two workers are polling the same work pool, either one can claim a scheduled run and then create the Kubernetes job/pod for it. So the behavior is roughly: - both workers poll the pool - Prefect hands a run to one of them - that worker submits the Kubernetes pod/job - the other worker does not also submit it So there is work distribution, but it’s not “load balancing” in the classic Kubernetes service/router sense. It’s more like: - competing consumers on the same queue Important implications - runs are not duplicated - either worker may pick up a given job - having two workers gives redundancy - throughput can improve if one worker is busy or unavailable How selection works in practice It’s not usually something you control run-by-run. The worker that successfully claims the run gets it. So if both workers are healthy: - worker A might submit one run - worker B might submit the next - or one may submit several in a row depending on timing/capacity For Kubernetes work pools specifically Once a worker claims a run, it creates the Kubernetes Job using the work pool/job template. So yes, either worker can be the one that spins up the pod. Things to be aware of - both workers should be on the same compatible version/config - both workers should have the same access/permissions in EKS - if one worker is misconfigured, runs it claims may fail even if the other one is healthy - if you want predictable behavior, keep the workers homogeneous So the short answer is: Yes — with two workers on the same work pool, either worker can pick up and launch the job pod. Docs: - Workers - Work pools
2 Views
Open in Slack
PreviousNext
Powered by