Thanks for the detailed context — that helps a lot. Short answer This almost always comes down to Python auto‑instrumentation not actually patching

requests

in the flow-run container on Kubernetes. The two most common causes are: - The OTel Operator injected the sidecar/SDK, but did not inject Python auto‑instrumentation into the actual Prefect job container (or injected into the wrong container name). - A version mismatch between your image’s `requests`/`urllib3` and the Operator’s Python autoinstrumentation image so the

requests

patch never applies. The

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

env var is unrelated to OpenTelemetry tracing. What to verify on your flow-run pod 1) Confirm Python auto‑instrumentation is injected into the Prefect job container - Your flow-run pod created by the Kubernetes work pool should have annotations like: -

<http://instrumentation.opentelemetry.io/inject-python|instrumentation.opentelemetry.io/inject-python>: "true"

-

<http://instrumentation.opentelemetry.io/container-names|instrumentation.opentelemetry.io/container-names>: "prefect-job"

← important: Prefect’s main container name is typically

prefect-job

- In the container spec for

prefect-job

, you should see either: - The entrypoint/command prefixed with

opentelemetry-instrument …

, or -

PYTHONPATH

includes the OTel sitecustomize path (commonly something like

/otel-auto-instrumentation

), which ensures auto‑patching at interpreter start. If you don’t see those in the

prefect-job

container, the Operator is not actually auto‑instrumenting Python there — it’s likely just attaching the sidecar/collector. 2) Check versions inside the running flow pod There have been real incompatibilities where older

opentelemetry-instrumentation-requests

could not patch newer `requests`/`urllib3` (e.g., urllib3 v2). Exec into the

prefect-job

container and run:

python - <<'PY'
import requests, urllib3
print("requests:", requests.__version__, "urllib3:", urllib3.__version__)
try:
    import opentelemetry.instrumentation.requests as r
    print("otel-instrumentation-requests module present:", r.__file__)
except Exception as e:
    print("otel-instrumentation-requests import failed:", e)

# Sanity check: did the instrumentation run?
from opentelemetry.instrumentation.requests import RequestsInstrumentor
print("requests already instrumented:", getattr(RequestsInstrumentor(), "_instrumented", False))
PY

- If

opentelemetry.instrumentation.requests

isn’t importable in the app container, the Operator didn’t inject Python auto‑instrumentation correctly. - If your

requests/urllib3

are very new and the injected OTel instrumentation is old, upgrade the Operator’s Python autoinstrumentation image to a recent version or pin

requests/urllib3

to a compatible set. 3) Ensure you’re not spawning new Python processes for tasks If you’re using a process‑based task runner (e.g., running tasks in subprocesses), the child process may start without the auto‑instrumentation bootstrap. The default Prefect 3 task runner is thread‑based, so this usually isn’t an issue unless you changed it. If you are using processes, you may need to: - Ensure the child processes start under

opentelemetry-instrument

, or - Manually instrument

requests

early in your code (see workaround below). How to fix in Prefect/Kubernetes - Make sure the OTel annotations are applied to the flow-run pods (not just the worker Deployment). In a Kubernetes work pool, add these to your base job template metadata:

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    <http://instrumentation.opentelemetry.io/inject-python|instrumentation.opentelemetry.io/inject-python>: "true"
    <http://instrumentation.opentelemetry.io/container-names|instrumentation.opentelemetry.io/container-names>: "prefect-job"
spec:
  template:
    spec:
      containers:
      - name: prefect-job
        # your image/command/env here

- If you control the OTel Operator Instrumentation resource, point it to a recent Python autoinstrumentation image so it includes a current

opentelemetry-instrumentation-requests

that supports your

requests/urllib3

versions.

- If needed, pin `requests`/`urllib3` in your image to versions known to work with your Operator’s instrumentation bundle. Quick, safe workaround (to confirm root cause) Add this at the very start of your flow module (before you import or use

requests

anywhere else):

try:
    from opentelemetry.instrumentation.requests import RequestsInstrumentor
    RequestsInstrumentor().instrument()
except Exception as e:
    print("Failed to instrument requests:", e)

If spans start showing up after this change, it confirms the auto‑instrumentation was not properly patching

requests

in the pod. Notes - It’s expected that you see OTel logs/spans for other things while

requests

remains uninstrumented if only that instrumentation failed to load or patch. -

PREFECT_CLOUD_ENABLE_ORCHESTRATION_TELEMETRY=false

only disables Prefect’s own product telemetry — it does not impact OpenTelemetry instrumentation. If you can share: - The flow-run pod’s annotations and the container spec for

prefect-job

- The output of the version checks above - Your Kubernetes work pool base job template …I can help pinpoint which of the above is the culprit and suggest the minimal change.

Good sleuthing — having the

opentelemetry-instrument

binary in PATH and being able to import

opentelemetry.instrumentation.requests

proves the bits are present in the container, but it doesn’t prove that auto‑instrumentation is actually being activated for the

prefect flow-run execute

process. In practice, this usually comes down to one of these in flow-run pods: - The Operator did not actually wrap the

prefect-job

container’s command with

opentelemetry-instrument

and/or its

sitecustomize

is not being loaded at interpreter start. - A different

sitecustomize

is shadowing the OTel one, so auto‑instrumentation never runs. - Instrumentation for

requests

is disabled via env, or there’s a version mismatch with

requests/urllib3

so the patch silently fails. Targeted checks to run in the flow-run pod 1) Confirm the OTel sitecustomize is the one being loaded

python - <<'PY'
import os, sys, inspect
print("PYTHONPATH=", os.environ.get("PYTHONPATH"))
print("OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=", os.environ.get("OTEL_PYTHON_DISABLED_INSTRUMENTATIONS"))
print("OTEL_PYTHON_LOG_LEVEL=", os.environ.get("OTEL_PYTHON_LOG_LEVEL"))
print("sitecustomize loaded at startup:", 'sitecustomize' in sys.modules)
try:
    import sitecustomize
    print("sitecustomize file:", inspect.getfile(sitecustomize))
except Exception as e:
    print("import sitecustomize failed:", e)
PY

- You want

sitecustomize file:

to point to the OTel auto‑instrumentation path (commonly something like

/otel-auto-instrumentation/.../sitecustomize.py

). If it points elsewhere or isn’t loaded, auto‑instrumentation didn’t fire. 2) Check requests/urllib3 and OTel instrumentation versions

python - <<'PY'
import requests, urllib3
print("requests:", requests.__version__, "urllib3:", urllib3.__version__)
try:
    import opentelemetry.instrumentation.requests as r
    import pkgutil, opentelemetry
    print("otel-instrumentation-requests present:", r.__file__)
    print("opentelemetry core:", opentelemetry.__version__)
except Exception as e:
    print("import otel instrumentation failed:", e)
PY

- If you have urllib3 v2.x with an older OTel instrumentation bundle, the patch can fail silently. Upgrading the Operator’s Python auto‑instrumentation image (or pinning

requests/urllib3

to a compatible set) fixes this. 3) Enable OTel debug logs to see if requests was instrumented Set in the pod env and re-run:

OTEL_PYTHON_LOG_LEVEL=debug

You should see messages like “Instrumented requests” or a failure reason. Kubernetes specifics to double‑check - Ensure the instrumentation annotations are applied to the flow-run Pod template and target the

prefect-job

container: -

<http://instrumentation.opentelemetry.io/inject-python|instrumentation.opentelemetry.io/inject-python>: "true"

-

<http://instrumentation.opentelemetry.io/container-names|instrumentation.opentelemetry.io/container-names>: "prefect-job"

- If the Operator relies on sitecustomize (vs. wrapping the command), make sure

/otel-auto-instrumentation

is first in

PYTHONPATH

. If another

sitecustomize

is found earlier on

sys.path

, OTel’s will be shadowed and no auto‑instrumentation will occur. - Verify

OTEL_PYTHON_DISABLED_INSTRUMENTATIONS

does not include

requests

. Two reliable fixes - Preferred: explicitly wrap the Prefect command in your Kubernetes work pool’s base job template so you don’t rely on

sitecustomize

ordering:

apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: prefect-job
        command: ["opentelemetry-instrument", "python", "-m", "prefect", "flow-run", "execute"]
        # ... your image/env/args here

This guarantees the instrumentation bootstrap runs before your flow code executes. - Or, ensure the Operator’s injected

/otel-auto-instrumentation

directory is the first entry in

PYTHONPATH

for the

prefect-job

container so its

sitecustomize

is the one that loads at interpreter startup. Quick confirmation workaround in code

If you want to confirm the root cause, instrument

requests

manually at the very top of your flow module:

try:
    from opentelemetry.instrumentation.requests import RequestsInstrumentor
    RequestsInstrumentor().instrument()
except Exception as e:
    print("Failed to instrument requests:", e)

If spans appear after this change, it confirms the auto‑instrumentation bootstrap did not run in the flow-run process. Why this points away from Prefect

prefect flow-run execute

does not unset OTel env or modify

PYTHONPATH

. When this symptom appears only in Kubernetes pods, it’s almost always: - Injection targeting the wrong container name, or -

sitecustomize

not being loaded (or being shadowed), or - An instrumentation/requests compatibility issue. If you can share: - The flow-run Pod annotations and the

prefect-job

container spec (command/args/env) - Output of the sitecustomize and version checks above …I can pinpoint exactly which of the above is happening and suggest the minimal change.

I hear you. Given your checks, the missing requests spans point to the flow-run subprocess not being started under the OTel bootstrap. Key detail: when a flow runs via Kubernetes, Prefect launches your code in a fresh Python process: - The container starts the parent process (your container entrypoint, typically “python -m prefect …”). - That parent then spawns a child process “python -m prefect.engine” to actually execute the flow. - If your OTel setup relies on wrapping the entrypoint with

opentelemetry-instrument

(or a wrapper script) and does not also ensure the OTel sitecustomize/PYTHONPATH is present for all descendant processes, the child process will not be auto-instrumented — which perfectly explains “OTel is present and exporting logs” while “requests spans are missing”. What to verify quickly inside the flow pod Run this to confirm whether the auto-instrumentation bootstrap (sitecustomize) is what the child process would see:

python - <<'PY'
import os, sys, inspect
print("PYTHONPATH=", os.environ.get("PYTHONPATH"))
print("sitecustomize loaded at startup:", 'sitecustomize' in sys.modules)
try:
    import sitecustomize
    print("sitecustomize file:", inspect.getfile(sitecustomize))
except Exception as e:
    print("import sitecustomize failed:", e)

# Check if requests was actually patched
from opentelemetry.instrumentation.requests import RequestsInstrumentor
print("requests already instrumented:", getattr(RequestsInstrumentor(), "_instrumented", False))

import requests, urllib3
print("requests:", requests.__version__, "urllib3:", urllib3.__version__)
PY

- If

sitecustomize

is not loaded or points somewhere other than the OTel auto-instrumentation path, the child process will start uninstrumented. - If

_instrumented

is False, requests has not been patched in this interpreter. Why this shows up only in Kubernetes - Locally, you’re likely running your script under

opentelemetry-instrument

or an environment where sitecustomize is active for that single Python process. - In k8s, Prefect’s child process is a new interpreter. If your OTel Operator configuration only wraps the container’s entrypoint (and does not ensure the OTel sitecustomize/PYTHONPATH/env applies to all child processes), requests instrumentation won’t run in the child. Two reliable ways to fix 1) Wrap the Prefect command in your work pool’s base job template so the OTel env (including PYTHONPATH/sitecustomize) is present for the entire container process tree:

apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - name: prefect-job
        command: ["opentelemetry-instrument", "python", "-m", "prefect", "flow-run", "execute"]
        # args, env, image as you have them

This ensures the child “python -m prefect.engine” inherits the OTel bootstrap env and requests gets patched. 2) Or ensure the Operator injects PYTHONPATH so the OTel sitecustomize is first on sys.path for the prefect-job container. That way any new Python process (including the Prefect child) auto-loads OTel before requests is imported. Additional checks that often surface the root cause - Enable OTel debug logs in the pod: -

OTEL_PYTHON_LOG_LEVEL=debug

You should see “Instrumented requests” or a reason it was skipped/failed. - Confirm compatibility: old

opentelemetry-instrumentation-requests

+ new

urllib3

(v2.x) can silently fail to patch. If the pod shows urllib3 2.x, make sure your OTel Python auto-instrumentation bundle is recent enough, or pin `urllib3<2`/align versions. If you can share: - The prefect-job container’s command/args from the pod spec - The output of the snippet above (especially the sitecustomize path and

_instrumented

flag) - Your work pool base job template …I can point to the exact knob to turn. But the minimal change that fixes this in practice is wrapping the container command with

opentelemetry-instrument

so the child flow-run process starts under the same OTel bootstrap.

I get why this is frustrating. The behavior you’re seeing is consistent with how Prefect executes flows: your container’s entrypoint runs

prefect flow-run execute

(parent), which then spawns a new Python process to run your flow code (

python -m prefect.engine

, child). Auto-instrumenting the parent does not guarantee the child is instrumented. What’s likely happening in your case - You’re now wrapping the parent with

uv run opentelemetry-instrument prefect flow-run execute

. - Prefect then starts a fresh child process with

python -m prefect.engine

. - That child is not started under the

opentelemetry-instrument

bootstrap and is not running inside

uv run

. If the OTel auto-instrumentation relies on the bootstrap (sitecustomize on PYTHONPATH) provided by `opentelemetry-instrument`/uv rather than being injected at the container level, the child will start without the

sitecustomize

and

requests

will not be patched. - This explains why: -

uv run python -c "import sitecustomize"

works in your shell (a new uv-run process). - But inside the flow’s child process,

sitecustomize

is not loaded and

RequestsInstrumentor()._instrumented

is False. Important: Prefect does not strip PYTHONPATH or use `-S`; it builds the child env and merges

os.environ

from the parent. However, relying on

uv run

+

opentelemetry-instrument

for the bootstrap is brittle for descendants, especially if the OTel

sitecustomize

lives in a uv ephemeral env and the child ends up using the system interpreter. The most reliable fix Run the engine directly under OTel instrumentation so the actual interpreter that executes your flow starts with the bootstrap: - In your Kubernetes work pool base job template, change the

prefect-job

container command to:

command: ["opentelemetry-instrument", "python", "-m", "prefect.engine"]

- Do not wrap the CLI (

prefect flow-run execute

); invoke the engine directly under

opentelemetry-instrument

. This removes the parent/child mismatch and guarantees

requests

is patched. If you must keep

uv

- Prefer baking OTel auto-instrumentation into the same Python environment your container uses at runtime instead of relying on

uv run

for the bootstrap. - If you insist on uv, mirror the same approach but be explicit:

command: ["uv", "run", "opentelemetry-instrument", "python", "-m", "prefect.engine"]

Note: Even then, ensure the child uses the same interpreter and that the OTel

sitecustomize

path is visible to it (this is where uv can still be tricky). Or, rely on the Operator’s container-level injection - Use the OpenTelemetry Operator’s Python injection on the

prefect-job

container (not just sidecars), so it sets PYTHONPATH for the whole container. That way any Python process, parent or child, loads the OTel

sitecustomize

automatically. - Annotations to include on the flow-run pod template:

<http://instrumentation.opentelemetry.io/inject-python|instrumentation.opentelemetry.io/inject-python>: "true"
<http://instrumentation.opentelemetry.io/container-names|instrumentation.opentelemetry.io/container-names>: "prefect-job"

This avoids depending on

opentelemetry-instrument

entirely. How to confirm quickly - After switching the command to

opentelemetry-instrument python -m prefect.engine

, enable debug:

OTEL_PYTHON_LOG_LEVEL=debug

You should see log lines indicating “Instrumented requests”. - You can also inspect the engine process env in the pod:

ps -ef | grep prefect.engine
# take the PID of python -m prefect.engine
tr '\0' '\n' </proc/<PID>/environ | egrep 'PYTHONPATH|OTEL|VIRTUAL_ENV|UV_'

- And verify at runtime by adding a tiny probe at the start of your flow file:

from opentelemetry.instrumentation.requests import RequestsInstrumentor
print("requests instrumented at start:", getattr(RequestsInstrumentor(), "_instrumented", False))

Version sanity checks (just in case)

- Python 3.12 requires recent OTel packages. Ensure you’re on a current OTel Python auto-instrumentation bundle and

opentelemetry-instrumentation-requests

that’s compatible with your `requests`/`urllib3` versions (urllib3 2.x requires newer instrumentation). Bottom line This isn’t Prefect “blocking” OTel — it’s the extra subprocess boundary. Starting the engine directly under

opentelemetry-instrument

(rather than the CLI) is the clean, deterministic way to ensure the process that executes your flow actually auto-instruments

requests

.