vinitv
04/29/2026, 9:50 AMMarvin
04/29/2026, 9:50 AMMarvin
04/29/2026, 9:53 AMDT_PYTHON_OPTIONS is not something Prefect normally invents on its own.
Most likely, it is being injected by your environment/image/runtime and then passed into the Docker job configuration that Prefect uses for the container. In your case, that strongly suggests a Dynatrace/OneAgent-style instrumentation variable, since nativeLibPath=...liboneagentpython.so is exactly that kind of setting.
What’s probably happening
- On the VSI process worker, your EPV CLI sees a library path that works, like /lib64/...
- On the Docker worker, the flow runs inside a container, and DT_PYTHON_OPTIONS points to /app/gtdt/...
- EPV appears to also inspect/use that setting, so inside the container it tries to use the wrong .so location and fails with PDKTC511E
So the root issue is likely environment inheritance / injection, not Prefect-specific EPV logic.
Who is setting it?
Usually one of these:
1. A host-level agent/injector on the VSI
2. Your Docker image via ENV DT_PYTHON_OPTIONS=...
3. The Prefect work pool base job template or deployment job_variables.env
4. A container runtime wrapper / entrypoint script
Prefect’s Docker worker does set an environment dict for the container, but DT_PYTHON_OPTIONS is not a standard Prefect variable.
What to check first
1. Inspect the work pool:
bash
prefect work-pool inspect <your-pool-name> --output json
Look in:
- base_job_template.job_configuration.env
2. Inspect the deployment:
bash
prefect deployment inspect "<flow-name>/<deployment-name>"
3. Inspect the image:
bash
docker inspect <your-image> | jq '.[0].Config.Env'
4. Check the worker host:
bash
env | grep DT_PYTHON_OPTIONS
If it exists on the host or image, that’s your source.
How to resolve it
You have a few options:
Option 1: Explicitly unset or override DT_PYTHON_OPTIONS for the Docker job
If EPV should not see that variable in the container, remove it from the container env.
For a deployment, set job_variables.env so it overrides the inherited value.
Example idea:
python
my_flow.from_source(
source=".",
entrypoint="flows.py:my_flow",
).deploy(
name="my-deployment",
work_pool_name="docker-pool",
job_variables={
"env": {
"DT_PYTHON_OPTIONS": ""
}
}
)
If empty string is not enough for EPV, you may need to ensure it is absent entirely by removing it from the work pool base job template instead.
Option 2: Set it to the correct in-container path
If EPV does need the variable, point it to the correct library path inside the container, e.g. /lib64/... if that path exists in the image.
python
job_variables={
"env": {
"DT_PYTHON_OPTIONS": "nativeLibPath=/lib64/liboneagentpython.so"
}
}
Only do this if that path is actually valid inside the container.
Option 3: Build/fix the Docker image so the expected library exists at the path
If /app/gtdt/... is expected by your org’s instrumentation, then the image may need the agent files mounted or baked in correctly.
Option 4: Disable the monitoring injection for this container/workload
If this is Dynatrace/OneAgent auto-injection, your infra team may need to exclude these Prefect-run containers or adjust injection behavior.
My recommendation
Given your description, I’d try this order:
1. Find where DT_PYTHON_OPTIONS is coming from
2. For the Docker deployment, override or remove it
3. If EPV requires it, set it explicitly to the correct in-container path
4. If the variable keeps reappearing, investigate Dynatrace/agent injection on the VSI or in the image
Important detail
A path that works on the VSI host does not automatically work in the Docker container. /lib64 on the host and /lib64 in the container are different filesystems unless you deliberately mount/share them.
Useful commands
bash
prefect work-pool inspect <pool-name> --output json
```bashMarvin
04/29/2026, 9:53 AM```bash
docker inspect <image-name>
bash
docker exec <container-id> env | grep DT_PYTHON_OPTIONS
Docs:
- Work pools
- Deployments
If you want, I can help you identify the exact field to change if you paste:
- your Docker work pool base_job_template
- and/or the deployment job_variables / prefect.yaml snippetvinitv
05/18/2026, 12:36 PMMarvin
05/18/2026, 12:36 PMMarvin
05/18/2026, 12:38 PM1, the worker should mark the flow run as Crashed / failed infrastructure, not leave it in Running.
So if it stays Running, the usual issue is not the exit code itself — it’s that the worker or the flow process could not report the terminal state back to the API.
Most likely causes:
* the container lost connectivity to the Prefect API
* the worker lost connectivity to the Prefect API
* the worker crashed or hung after the container exited
* the container was removed too quickly (auto_remove) before the worker finished inspecting it
* Docker daemon/API became unresponsive while the worker was waiting on the container
What Prefect does internally:
* Docker worker reads container exit code
* if non-zero, it proposes a crashed state to the API
* if that API update never succeeds, the run can remain stuck in Running
A few concrete things to check:
1. Worker logs
Look for errors like:
* Failed to update state of flow run
* Docker API errors / timeouts
* container removed before wait completed
* API/network connection errors
Run worker with debug logging:
bash
export PREFECT_LOGGING_LEVEL=DEBUG
prefect worker start --pool <your-pool>
2. Flow container logs
Check whether the container can reach your Prefect API URL. This is a very common root cause.
In particular verify:
* PREFECT_API_URL is correct inside the container
* if self-hosted, don’t rely on localhost unless the API is actually inside that same container
* on Linux Docker setups, host.docker.internal may need special handling depending on your network setup
Related issue about container/API connectivity:
Docker agent with server tasks stuck submitted: host.docker.internal connection error
3. Disable auto-removal temporarily
If your pool/job config removes containers immediately, disable that so you can inspect the finished container:
yaml
job_variables:
auto_remove: false
stream_output: true
4. Increase Docker client timeout
If Docker is slow/unresponsive:
bash
export DOCKER_CLIENT_TIMEOUT=600
5. Enable zombie/unresponsive run detection
Prefect recommends heartbeat-based detection for runs that get stuck in Running when infrastructure disappears or cannot report back.
Docs:
Detect zombie flows
You can also enable flow heartbeats with:
bash
export PREFECT_FLOWS_HEARTBEAT_FREQUENCY=180
Then set that in the worker/container environment too.
Important note:
This symptom is often a network/reporting problem, not a Docker execution problem. The container may have exited correctly with 1, but if neither the flow process nor the worker can successfully call the API afterward, the run can remain Running.
If you want, I can help you narrow it down quickly if you share:
* your Prefect version
* whether this is Prefect Cloud or self-hosted server
* the worker logs around the time the container exited
* whether auto_remove is enabled
* the container/job config for PREFECT_API_URLvinitv
05/18/2026, 1:08 PMMarvin
05/18/2026, 1:08 PMMarvin
05/18/2026, 1:11 PM1
* worker tries to report terminal/crashed state
* if API/network call fails at that moment, Prefect retries for a short window
* if it still fails, that update is dropped
* the flow run can remain stuck in Running even though the worker is healthy afterward and can run new flows
So a later return to normal connectivity does not automatically fix that already-stuck run.
Details:
* default client retries are about 5 retries
* backoff is exponential, roughly over about a minute total
* after that, worker logs something like Failed to update state of flow run ...
* there is no background reconciliation loop that later says “oh, that container already exited, let me mark it crashed now”
This matches what you observed:
* same worker can run flows successfully now
* old run is still stuck
* container still exists
=> that strongly suggests the failure happened during the terminal state report window, not that the worker is permanently broken
What I’d check next:
1. Worker logs around container exit time
Search for:
text
Failed to update state of flow run
httpx
ConnectError
ReadTimeout
LocalProtocolError
RemoteProtocolError
2. Container logs
If the flow process inside the container also failed to talk to the API near shutdown, that can contribute too.
3. Prefect version
There have been various client/network edge cases over time, so if you are not on a recent 3.x, upgrading is worth considering.
4. Enable zombie detection / heartbeats
This is the protection against exactly these “run stuck in Running” cases:
Detect zombie flows
If you want, paste:
* your Prefect version
* the worker log lines from just before/after the container exited
* whether you use Cloud or self-hosted API
and I can help pinpoint whether it was an API retry exhaustion, Docker wait issue, or something else.vinitv
06/03/2026, 9:47 AMMarvin
06/03/2026, 9:47 AMMarvin
06/03/2026, 9:50 AMprefect-redis==0.2.10, the cancellation/cleanup listener uses Redis in a way that is more sensitive than simple direct commands. When you connect straight to a single Redis instance, the connection stays consistent and everything works. When you put HAProxy in front, the listener can hit connection/proxy behavior that does not affect simpler request/response usage.
What’s probably happening:
1. Long-lived Redis connection behavior
The cancellation listener uses long-lived Redis client connections and coordination patterns that are sensitive to disconnects/resets. HAProxy can interrupt or recycle these connections because of:
* idle timeout
* server timeout
* health-check-driven backend changes
* TCP connection reuse behavior
So even if normal Redis calls work, this listener can fail because it depends on a stable connection over time.
2. Transaction / connection-affinity issues
prefect-redis cleanup/cancellation internals use Redis transaction-style coordination (WATCH / MULTI patterns). Those patterns are connection-specific. If a proxy causes the effective backend connection/context to change unexpectedly, Redis transaction state is lost or invalidated.
That’s why:
* direct single Redis => stable, works
* HAProxy in front => intermittent connection/protocol errors in listener
3. Blocking / listener-style operations are more proxy-sensitive
Listener/consumer loops often use blocking or near-blocking reads/polls. These are much more likely than ordinary Redis GET/SET traffic to expose HAProxy timeout problems.
So this is not surprising if you only see it in the cancellation listener.
What to check in HAProxy first:
* increase timeouts substantially
text
timeout connect 10s
timeout client 300s
timeout server 300s
timeout tunnel 300s
timeout client-fin 300s
timeout server-fin 300s
* if you use health checks / balancing, make sure HAProxy is not bouncing these long-lived Redis connections between backends
* if this is actually multiple Redis backends behind HAProxy, that is a big red flag unless you have a very specific Redis-aware architecture
* prefer pure TCP mode, not anything Redis-protocol-aware unless intentionally configured
Important question:
Are you using HAProxy in front of a single Redis server, or load balancing across multiple Redis servers?
Because:
* single Redis behind HAProxy => usually a timeout / connection reset / keepalive issue
* multiple Redis servers behind HAProxy => very likely incompatible with the listener’s transaction/connection assumptions
Best practical recommendation:
For Prefect cancellation/cleanup Redis, connect directly to Redis if possible instead of through HAProxy, especially with prefect-redis 0.2.10.
If you must keep HAProxy:
* use TCP passthrough
* pin to a single backend
* increase idle/server/client timeouts
* avoid balancing across multiple standalone Redis instances for this workload
If you want, paste:
* the exact Redis error from the cancellation listener
* your HAProxy backend/frontend config
* whether HAProxy points to one Redis or several
and I can tell you which of those failure modes you’re hitting.vinitv
06/03/2026, 10:19 AMMarvin
06/03/2026, 10:19 AMMarvin
06/03/2026, 10:20 AMvinitv
06/03/2026, 10:37 AMMarvin
06/03/2026, 10:37 AMMarvin
06/03/2026, 10:39 AMaws_credentials rather than trying to force a fake value
* You set the cluster explicitly to the cluster name/ARN you want the worker to submit tasks into
Example base job template:
json
{
"job_configuration": {
"region": "{{ region }}",
"cluster": "{{ cluster }}",
"launch_type": "{{ launch_type }}",
"family": "{{ family }}",
"image": "{{ image }}",
"cpu": "{{ cpu }}",
"memory": "{{ memory }}",
"execution_role_arn": "{{ execution_role_arn }}",
"task_role_arn": "{{ task_role_arn }}",
"configure_cloudwatch_logs": true,
"cloudwatch_logs_options": {
"awslogs-region": "{{ region }}",
"awslogs-group": "{{ cloudwatch_logs_group }}",
"awslogs-stream-prefix": "prefect"
},
"env": "{{ env }}",
"command": "{{ command }}",
"stream_output": true,
"vpc_id": "{{ vpc_id }}",
"network_configuration": "{{ network_configuration }}"
},
"variables": {
"type": "object",
"properties": {
"region": {
"type": "string",
"title": "AWS Region",
"default": "us-east-1"
},
"cluster": {
"type": "string",
"title": "ECS Cluster",
"default": "your-ecs-cluster-name"
},
"launch_type": {
"type": "string",
"title": "Launch Type",
"default": "FARGATE"
},
"family": {
"type": "string",
"title": "Task Definition Family",
"default": "prefect-flow"
},
"image": {
"type": "string",
"title": "Image",
"default": "your-image:latest"
},
"cpu": {
"type": "string",
"title": "CPU",
"default": "1024"
},
"memory": {
"type": "string",
"title": "Memory",
"default": "2048"
},
"execution_role_arn": {
"type": "string",
"title": "Execution Role ARN",
"default": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
},
"task_role_arn": {
"type": "string",
"title": "Task Role ARN",
"default": "arn:aws:iam::123456789012:role/prefectFlowTaskRole"
},
"cloudwatch_logs_group": {
"type": "string",
"title": "CloudWatch Logs Group",
"default": "/ecs/prefect-flows"
},
"env": {
"type": "object",
"title": "Environment Variables",
"default": {}
},
"command": {
"type": "string",
"title": "Command",
"default": "python -m prefect.engine"
},
"vpc_id": {
"type": "string",
"title": "VPC ID",
"default": ""
},
"network_configuration": {
"type": "object",
"title": "Network Configuration",
"default": {
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"subnets": [
"subnet-xxxxxxxx"
],
"securityGroups": [
"sg-xxxxxxxx"
]
}
}
}
}
}
}
Important part for your requirement:
there is no aws_credentials section at all
That means Prefect will use the normal AWS credential chain, and because your worker is running as an ECS task, it will use the task role credentials automatically.
If you want the worker to submit runs into the same cluster, set:
json
"cluster": "{{ cluster }}"
and give cluster the default value of your ECS cluster name or ARN.
If you want something a bit more practical/minimal, here is a cleaner version you can start from:
```json
{Marvin
06/03/2026, 10:39 AMexecution_role_arn is for ECS task startup things like pulling from ECR / pushing logs
* task_role_arn is the IAM role the flow run container uses
* the worker task role must itself have permission to call ECS APIs like RunTask, DescribeTasks, RegisterTaskDefinition, etc.
* if your flow image is in ECR and logs go to CloudWatch, those roles must allow that too
Also, one correction to avoid an old 2.x pattern:
if you want to deploy in Prefect 3.x, use flow.from_source(...).deploy(...) or prefect deploy, not Deployment.build_from_flow().
Docs:
* Prefect AWS ECS worker docs
* Work pools
If you want, I can also give you:
* a full base job template exactly matching the current ECS worker schema, or
* a prefect work-pool create --type ecs + update JSON workflow, or
* a Terraform-ready version of this template.vinitv
06/09/2026, 10:24 AMMarvin
06/09/2026, 10:24 AMMarvin
06/09/2026, 10:24 AM3.6.27 that I can confirm from the latest release notes I have access to.
These are from the newer release notes currently available, and include fixes after `3.6.27`:
Bug fixes
* Fix deploy init field parsing — #22183
* Fix parse_obj_as() crash with StopIteration on empty dict — #22188
* Fix prompt to save new deployments into an existing prefect.yaml — #22195
* Filter stdlib paths from PYTHONPATH to prevent subprocess CLI breakage — #22179
* Recover perpetual services after Redis disruption — #22213
A few especially relevant ones based on your earlier questions:
* Recover perpetual services after Redis disruption — likely relevant if you were seeing Redis/cancellation-listener warnings
* deploy/prefect.yaml fixes — relevant if you use prefect deploy
Full latest changelog:
Prefect 3.7.3...3.7.4 changelog
If you want, I can also give you one of these:
* all bug fixes between 3.6.27 and latest, grouped by version
* just Redis/worker/cancellation-related fixes after 3.6.27
* whether upgrading from 3.6.27 would likely help your specific issuevinitv
06/11/2026, 7:11 AMMarvin
06/11/2026, 7:11 AMMarvin
06/11/2026, 7:14 AMtext
http://<your-host>:4200
and the API is typically:
text
http://<your-host>:4200/api
Check health:
bash
curl http://<your-host>:4200/api/health
2. Point your client to your self-hosted API
Set:
bash
prefect config set PREFECT_API_URL="http://<your-host>:4200/api"
Then confirm:
bash
prefect config view
3. Do not log into Prefect Cloud for this server
If your browser has an existing Prefect Cloud session, the UI can get confused and show Cloud-style workspace messaging.
Try:
* open the self-hosted UI in an incognito/private window
* log out of Prefect Cloud
* clear site storage/cookies for Prefect UI
* then reopen your on-prem URL directly
4. Check reverse proxy / ingress config
If you have nginx/ingress in front of Prefect, make sure it is not rewriting requests to Prefect Cloud endpoints and that the UI can access:
* /api/health
* /ui-settings
For example:
bash
curl http://<your-host>:4200/ui-settings
You should get JSON for your local server settings.
Important point
Self-hosted Prefect Server does not use Cloud workspaces the same way Prefect Cloud does. There is no separate “grant me workspace access” step you need to perform for an on-prem install.
So the fix is usually:
use the self-hosted UI/API directly, not a Cloud URL/session
Useful docs:
* Self-host Prefect
* Prefect server concepts
If you want, send me:
* the exact URL you are opening in the browser
* your PREFECT_API_URL
* whether you put a reverse proxy in front of Prefect
and I can tell you exactly what is misconfigured.vinitv
06/11/2026, 7:15 AMMarvin
06/11/2026, 7:15 AMMarvin
06/11/2026, 7:15 AMPREFECT_API_URL alone — it is an Enterprise RBAC/workspace provisioning issue.
What to check with your Enterprise admin:
* Is there at least one workspace created?
* Is your user assigned to a workspace?
* If using SSO, are the IdP groups/claims mapped correctly?
* Are you logging in with the expected email/identity?
* Was initial Enterprise setup fully completed?
I can’t provision or assign the workspace for you, but the right path for Enterprise-specific setup/access issues is:
Prefect contact
If you want, I can help you narrow it down before you reach out — for example, if you tell me:
* are you using local auth or SSO?
* is this the first user logging in after install?
* do other users see the same message?
I can help you figure out whether it’s likely:
* missing workspace creation
* missing user assignment
* broken SSO group mapping
* wrong login identityvinitv
06/11/2026, 7:22 AMMarvin
06/11/2026, 7:22 AMMarvin
06/11/2026, 7:22 AMhelm, docker compose, terraform, etc.)
* auth mode (SSO, OIDC, local auth)
* whether all users see “no workspace access”
* whether any bootstrap/init job logs mention admin, org, workspace, or RBAC setup
Then use:
Prefect contact
If you want, I can help you draft a very short summary of the issue and the exact checks to perform in your deployment manifests/logs before you contact Prefect.vinitv
06/11/2026, 7:26 AMMarvin
06/11/2026, 7:26 AMMarvin
06/11/2026, 7:27 AMworkspace
* rbac
* auth
* oidc
* sso
* bootstrap
* admin
* claims
* groups
But I don’t want to tell you “set X in Y” without verified Enterprise docs, because that would be risky.
So short answer:
I don’t have a verified public location for that Enterprise bootstrap config.
Use:
Prefect contact
If you want, I can help you inspect your Helm values or deployment manifests and identify any likely bootstrap/auth settings.