<@ULVA73B9P> when running a kubernetes workpool wh...
# ask-marvin
a
@Marvin when running a kubernetes workpool where are the pull deployments steps run?
m
thought for 114.1 seconds
With a Kubernetes work pool, deployment pull steps run inside the Kubernetes pod/job created for the flow runnot on the worker. So the sequence is roughly: 1. The Kubernetes worker receives the flow run 2. The worker creates a Kubernetes Job/Pod 3. That pod starts
prefect flow-run execute ...
4. Inside that pod, Prefect prepares the workspace and executes the deployment’s
pull
steps 5. Then the flow code is imported and run Practical implication: anything needed by your pull steps must be available in the pod, for example: - git credentials - cloud storage credentials - network access to GitHub / S3 / GCS / internal artifact store - any env vars or secrets referenced by the pull step So if you’re using something like
git_clone
, the clone happens from the run pod itself, not from the worker host. Relevant docs: - Work pools - Code storage and retrieval - Workers If you want, I can also explain where
build
,
push
, and
pull
each run
in a Kubernetes deployment lifecycle, since that’s a very common point of confusion.
a
@Marvin in which directory is git code pulled to? how is the flow entrypoint resolved?
m
thought for 82.8 seconds
By default, git code is pulled into the current workspace root, and then Prefect may update the working directory based on the pull step’s output. For
git_clone
, the important detail is: - the clone happens in the pod’s current workspace - the step returns a
directory
- Prefect uses that returned
directory
as the working directory for resolving the entrypoint So effectively: - clone location: somewhere under the temporary workspace Prefect created for that flow run - working directory after clone: usually the cloned repo directory returned by
git_clone
Example:
Copy code
yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/my-org/my-repo.git>

entrypoint: flows/my_flow.py:my_flow
This behaves roughly like: 1. Prefect creates a temporary workspace dir 2.
git_clone
clones the repo into that workspace 3.
git_clone
returns something like
{"directory": "my-repo"}
4. Prefect sets working directory to
<workspace>/my-repo
5. Prefect resolves
flows/my_flow.py:my_flow
relative to that directory So the effective resolved file becomes:
Copy code
text
<workspace>/my-repo/flows/my_flow.py
How entrypoint resolution works If your entrypoint is a file path like:
Copy code
text
flows/my_flow.py:my_flow
Prefect resolves the file path relative to the final working directory selected by pull steps. If your entrypoint is already absolute, it uses that directly. If your pull steps change/select another directory, that directory becomes the base for entrypoint resolution. Practical rule of thumb - Make
entrypoint
relative to the root of the cloned repo - Let
git_clone
establish that repo as the working directory For example, if your repo looks like:
Copy code
text
my-repo/
  src/
    flows/
      etl.py
use:
Copy code
yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/my-org/my-repo.git>

entrypoint: src/flows/etl.py:my_flow
Docs: - Code storage and retrieval - Deployments If you want, I can also show you a full Kubernetes work pool deployment example with
git_clone
, including what env vars/secrets need to be on the pod.
a
@Marvin does setting PYTHONPATH in the container affect how the entrypoint is resolved?
m
thought for 128.7 seconds
Yes, but not usually in the way people first expect. Short version: - For a file entrypoint like
flows/my_flow.py:my_flow
, Prefect resolves the file path relative to the selected working directory from the pull/setup process.
PYTHONPATH
does not change that file path resolution. -
PYTHONPATH
can affect imports that happen after that file is loaded. - For a module-style entrypoint,
PYTHONPATH
matters more because Python module importing uses
sys.path
. For your case: file entrypoints If your deployment says:
Copy code
text
entrypoint: flows/my_flow.py:my_flow
Prefect first determines the working directory, typically from the pull step output, then resolves that to something like:
Copy code
text
<working_dir>/flows/my_flow.py
That resolution is based on the working directory, not on
PYTHONPATH
. After that, when Python executes
my_flow.py
, imports inside that file may be influenced by: - the script’s parent directory - the working directory -
PYTHONPATH
- normal site-packages So
PYTHONPATH
can help imports inside the flow code, but it does not make Prefect reinterpret
flows/my_flow.py
as a different file location. For module-style entrypoints If you were using a module import style instead, then
PYTHONPATH
matters more because Python is searching import paths to find the module. So conceptually: -
path/to/file.py:obj
→ file path resolution uses working directory -
package.module.obj
/ module-style loading → module discovery can depend on
PYTHONPATH
Practical takeaway If your flow isn’t being found: - check the working directory established by your pull steps - make sure
entrypoint
is relative to that directory If your flow file is found but imports inside it fail: - then
PYTHONPATH
may be relevant Example:
Copy code
yaml
pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/my-org/my-repo.git>

entrypoint: src/flows/etl.py:flow_fn
If the repo is cloned and the working directory becomes
<workspace>/my-repo
, Prefect looks for:
Copy code
text
<workspace>/my-repo/src/flows/etl.py
Setting:
Copy code
text
PYTHONPATH=/app
won’t change that file lookup. But it could help if
etl.py
contains imports that need
/app
on
sys.path
. If you want, I can help you reason through your exact repo layout and tell you what
entrypoint
and
PYTHONPATH
should be for that container.
a
@Marvin can you log the output of deployment pull steps?
m
thought for 116.0 seconds
Yes — partially. There are two kinds of “output” to think about for pull steps: 1. Prefect logger messages from the step 2. raw stdout/stderr produced by commands or subprocesses inside the step What you can see reliably Prefect does log pull step lifecycle messages, such as step execution / success / failure. Those are associated with deployment/workspace preparation. For built-in pull steps like
git_clone
, you should get useful log messages about what happened. For Kubernetes work pools specifically Since pull steps run inside the flow run pod, their output ends up in the pod logs. So if you want to inspect raw output from pull steps in Kubernetes, the most reliable place is: -
kubectl logs <pod-name>
- or whatever centralized logging your cluster uses Important limitation Raw
stdout/stderr
from pull-step subprocesses is not the same thing as normal flow/task logs in the Prefect UI. So: - pod/container logs → best place for raw pull-step output - Prefect UI → you may see higher-level step messages/events, but not always all raw shell output the way you might expect If you are using a shell-based pull/setup step, behavior also depends on the step: - some steps capture output as return values - some stream output to the runtime stderr/stdout - streamed output goes to infrastructure logs, not necessarily to Prefect API logs Practical answer If your goal is “I want to debug what my pull step is doing in a Kubernetes work pool,” then yes — check the Kubernetes pod logs. Example:
Copy code
text
kubectl logs <flow-run-pod-name> -n <namespace>
If your goal is “I want that output to appear clearly in Prefect logs/UI,” that’s more limited. A common workaround is to make your custom pull/setup logic emit explicit logs in a way Prefect captures, or move important diagnostics into the flow itself after startup. Docs: - Code storage and retrieval - Workers - Work pools If you want, I can show you a custom pull step pattern that makes debugging much easier in Kubernetes, including how to surface
pwd
,
ls
, and env diagnostics.