I'm running into an issue where around 30% of my p...
# ask-community
t
I'm running into an issue where around 30% of my prefect runs for one my deployments fails at the git_clone and set_working_directory step because the directory where the code would be cloned into does not exist. This is especially weird as it works most of the time, but I guess 30% of the time the clone fails? I tried adding a shell script to sleep in between the clone and set working directory in case it was due to some race condition from the cloning, to no avail. When I add a debug step to run
ls /
, on successful runs the code is cloned to the right place under
/code-main
and on failed runs that directory is not there. There is no logs from the git clone step to suggest that the clone fails. The gh token is valid and it does work most of the time. Any ideas? This is on Prefect Cloud using a Modal Work Pool
@Marvin is there any way to add debug logs to the deployment pull steps like prefect.deployments.steps.git_clone?
m
thought for 624.2 seconds
Yes. A few practical ways to get more visibility into pull steps (including
prefect.deployments.steps.git_clone
) in Prefect 3.x: 1) Raise Prefect’s log level (affects pull steps too) - Set this on the process that runs your worker (or in your work pool’s base job template env):
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
- Optionally scope extra loggers (useful if you only want verbose logs from pull steps):
Copy code
PREFECT_LOGGING_EXTRA_LOGGERS=prefect.deployments.steps
- Docs: Logging customization (Prefect 3) 2) Use a custom logging config (target just the pull-step logger) - Create a logging YAML and point Prefect to it:
Copy code
PREFECT_LOGGING_CONFIG_PATH=/path/to/logging.yml
- Minimal example to enable DEBUG for pull steps only:
Copy code
version: 1
formatters:
  standard:
    format: "%(asctime)s | %(levelname)s | %(name)s - %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    level: DEBUG
loggers:
  prefect.deployments.steps:
    level: DEBUG
    handlers: [console]
    propagate: True
root:
  level: INFO
  handlers: [console]
3) Get verbose output from Git itself -
git_clone
shells out to
git
, so you can enable Git tracing in the worker environment:
Copy code
GIT_TRACE=1
GIT_CURL_VERBOSE=1
- Note: these can include sensitive info in logs. Be cautious when using tokens and public logs. 4) Replace
git_clone
with an explicit shell step for maximum control - If you need to see every command and env while cloning, swap the pull step to
run_shell_script
and drive
git
yourself:
Copy code
pull:
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        set -euxo pipefail
        git --version
        git clone --verbose <https://github.com/owner/repo> app
        cd app
        git submodule update --init --recursive
      env:
        GIT_TRACE: "1"
        GIT_CURL_VERBOSE: "1"
Notes - There isn’t a dedicated CLI flag like
--log-level
on `prefect worker start`; the environment variables above are the supported way in 3.x. -
git_clone
does not accept a verbosity parameter; raising Prefect’s log level and/or enabling Git trace are the paths to more detail. If you can share how you’re running your worker (Docker, Kubernetes, VM, etc.), I can suggest the best place to set these env vars in your setup.
t
@Marvin I am using Prefect Cloud with a Modal work pool. how can I set the environment variable to make the deployment pull steps use DEBUG logging?
m
thought for 106.3 seconds
Yes—since you’re using a Modal work pool, the pull steps run inside the Modal job container, not in the worker process. Set the env vars on the job so they’re present at runtime. You have three good options: 1) Set env on the Modal work pool’s base job template (applies to all jobs from that pool) - In Prefect Cloud: Work Pools -> your Modal pool -> Edit base job template - Add under the template’s `env`:
Copy code
PREFECT_LOGGING_LEVEL: DEBUG
- Optional for git verbosity:
Copy code
GIT_TRACE: "1"
GIT_CURL_VERBOSE: "1"
2) Set env per deployment via job variables - In `prefect.yaml`:
Copy code
deployments:
  - name: my-deployment
    work_pool:
      name: my-modal-pool
    job_variables:
      env:
        PREFECT_LOGGING_LEVEL: DEBUG
        GIT_TRACE: "1"
        GIT_CURL_VERBOSE: "1"
- Or in Python when deploying:
Copy code
flow.from_source("<https://github.com/org/repo>", entrypoint="flows/flow.py:my_flow").deploy(
  name="my-deployment",
  work_pool_name="my-modal-pool",
  job_variables={
    "env": {
      "PREFECT_LOGGING_LEVEL": "DEBUG",
      "GIT_TRACE": "1",
      "GIT_CURL_VERBOSE": "1",
    }
  },
)
3) Use a custom logging config file (optional, for fine-grained control) - Bake a
logging.yml
into your Modal image or make it available at runtime, then set:
Copy code
PREFECT_LOGGING_CONFIG_PATH=/path/to/logging.yml
- Example to raise only pull-step logs:
Copy code
version: 1
formatters:
  standard:
    format: "%(asctime)s | %(levelname)s | %(name)s - %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    level: DEBUG
loggers:
  prefect.deployments.steps:
    level: DEBUG
    handlers: [console]
    propagate: True
root:
  level: INFO
  handlers: [console]
Notes -
PREFECT_LOGGING_LEVEL=DEBUG
is the simplest way to make
prefect.deployments.steps.git_clone
and other pull steps emit DEBUG logs. - Git tracing (
GIT_TRACE
,
GIT_CURL_VERBOSE
) is optional but helpful for diagnosing authentication/clone issues; be cautious as it can log sensitive details. Docs - Logging customization - Customize job variables - Using Modal with Prefect If you share how you’re creating/updating your deployment (prefect.yaml vs Python), I can provide the exact snippet for your setup.