Trey Gilliland
09/04/2025, 12:11 AMls /
, on successful runs the code is cloned to the right place under /code-main
and on failed runs that directory is not there.
There is no logs from the git clone step to suggest that the clone fails. The gh token is valid and it does work most of the time.
Any ideas? This is on Prefect Cloud using a Modal Work PoolTrey Gilliland
09/04/2025, 2:26 PMMarvin
09/04/2025, 2:26 PMMarvin
09/04/2025, 2:36 PMprefect.deployments.steps.git_clone
) in Prefect 3.x:
1) Raise Prefect’s log level (affects pull steps too)
- Set this on the process that runs your worker (or in your work pool’s base job template env):
PREFECT_LOGGING_LEVEL=DEBUG
- Optionally scope extra loggers (useful if you only want verbose logs from pull steps):
PREFECT_LOGGING_EXTRA_LOGGERS=prefect.deployments.steps
- Docs: Logging customization (Prefect 3)
2) Use a custom logging config (target just the pull-step logger)
- Create a logging YAML and point Prefect to it:
PREFECT_LOGGING_CONFIG_PATH=/path/to/logging.yml
- Minimal example to enable DEBUG for pull steps only:
version: 1
formatters:
standard:
format: "%(asctime)s | %(levelname)s | %(name)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
formatter: standard
level: DEBUG
loggers:
prefect.deployments.steps:
level: DEBUG
handlers: [console]
propagate: True
root:
level: INFO
handlers: [console]
3) Get verbose output from Git itself
- git_clone
shells out to git
, so you can enable Git tracing in the worker environment:
GIT_TRACE=1
GIT_CURL_VERBOSE=1
- Note: these can include sensitive info in logs. Be cautious when using tokens and public logs.
4) Replace git_clone
with an explicit shell step for maximum control
- If you need to see every command and env while cloning, swap the pull step to run_shell_script
and drive git
yourself:
pull:
- prefect.deployments.steps.utility.run_shell_script:
script: |
set -euxo pipefail
git --version
git clone --verbose <https://github.com/owner/repo> app
cd app
git submodule update --init --recursive
env:
GIT_TRACE: "1"
GIT_CURL_VERBOSE: "1"
Notes
- There isn’t a dedicated CLI flag like --log-level
on `prefect worker start`; the environment variables above are the supported way in 3.x.
- git_clone
does not accept a verbosity parameter; raising Prefect’s log level and/or enabling Git trace are the paths to more detail.
If you can share how you’re running your worker (Docker, Kubernetes, VM, etc.), I can suggest the best place to set these env vars in your setup.Trey Gilliland
09/04/2025, 6:45 PMMarvin
09/04/2025, 6:45 PMMarvin
09/04/2025, 6:47 PMPREFECT_LOGGING_LEVEL: DEBUG
- Optional for git verbosity:
GIT_TRACE: "1"
GIT_CURL_VERBOSE: "1"
2) Set env per deployment via job variables
- In `prefect.yaml`:
deployments:
- name: my-deployment
work_pool:
name: my-modal-pool
job_variables:
env:
PREFECT_LOGGING_LEVEL: DEBUG
GIT_TRACE: "1"
GIT_CURL_VERBOSE: "1"
- Or in Python when deploying:
flow.from_source("<https://github.com/org/repo>", entrypoint="flows/flow.py:my_flow").deploy(
name="my-deployment",
work_pool_name="my-modal-pool",
job_variables={
"env": {
"PREFECT_LOGGING_LEVEL": "DEBUG",
"GIT_TRACE": "1",
"GIT_CURL_VERBOSE": "1",
}
},
)
3) Use a custom logging config file (optional, for fine-grained control)
- Bake a logging.yml
into your Modal image or make it available at runtime, then set:
PREFECT_LOGGING_CONFIG_PATH=/path/to/logging.yml
- Example to raise only pull-step logs:
version: 1
formatters:
standard:
format: "%(asctime)s | %(levelname)s | %(name)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
formatter: standard
level: DEBUG
loggers:
prefect.deployments.steps:
level: DEBUG
handlers: [console]
propagate: True
root:
level: INFO
handlers: [console]
Notes
- PREFECT_LOGGING_LEVEL=DEBUG
is the simplest way to make prefect.deployments.steps.git_clone
and other pull steps emit DEBUG logs.
- Git tracing (GIT_TRACE
, GIT_CURL_VERBOSE
) is optional but helpful for diagnosing authentication/clone issues; be cautious as it can log sensitive details.
Docs
- Logging customization
- Customize job variables
- Using Modal with Prefect
If you share how you’re creating/updating your deployment (prefect.yaml vs Python), I can provide the exact snippet for your setup.