Ferchau Gupta
05/26/2026, 7:32 AMPREFECT__SERVER__DATABASE__CONNECTION_URL ? I am using Prefect 2.20Marvin
05/26/2026, 7:32 AMMarvin
05/26/2026, 7:34 AMbash
~/.prefect/prefect.db
There’s also some backward-compatibility logic: if prefect.db does not exist but ~/.prefect/orion.db does, Prefect may use orion.db instead.
Best ways to check
1. Show the active config, including defaults
bash
prefect config view --show-defaults --show-sources
Look for:
bash
PREFECT_API_DATABASE_CONNECTION_URL='sqlite+aiosqlite:///.../.prefect/prefect.db'
PREFECT_HOME='.../.prefect'
2. Check whether the DB files exist
bash
ls -lah ~/.prefect/prefect.db
ls -lah ~/.prefect/orion.db
3. Inspect it in Python
python
from prefect.settings import PREFECT_API_DATABASE_CONNECTION_URL, PREFECT_HOME
print("DB URL:", PREFECT_API_DATABASE_CONNECTION_URL.value())
print("PREFECT_HOME:", PREFECT_HOME.value())
That should print the resolved connection URL Prefect is using.
Important note on the env var name for Prefect 2.x
In Prefect 2.x, the setting you usually want to inspect is:
bash
PREFECT_API_DATABASE_CONNECTION_URL
not PREFECT__SERVER__DATABASE__CONNECTION_URL.
So you can also check directly:
bash
echo $PREFECT_API_DATABASE_CONNECTION_URL
env | grep ^PREFECT_
If you want, I can also show you how to check this from a running server process or help confirm whether your instance is using SQLite vs Postgres.Ferchau Gupta
05/28/2026, 6:25 AMjob_watch_timeout_seconds in Prefect 2.20, which defines the number of seconds for which a worker will observe a flow run?Marvin
05/28/2026, 6:25 AMMarvin
05/28/2026, 6:28 AMjob_watch_timeout_seconds is not a global worker setting.
It exists for Kubernetes workers / Kubernetes job infrastructure, and it controls how long the worker will keep watching the Kubernetes job before timing out.
For Kubernetes in Prefect 2.20
- `job_watch_timeout_seconds`: max time to watch the job overall
- `pod_watch_timeout_seconds`: max time to wait for pod creation
If job_watch_timeout_seconds is None (the default), the worker waits indefinitely.
Important: this is not a setting for all worker types.
Other worker types in 2.20
- Process worker: no equivalent watch-timeout setting
- Docker worker: no equivalent watch-timeout setting
- Vertex AI worker: has job_watch_poll_interval, but that is polling frequency, not a total timeout
- AWS ECS: uses different settings such as task_start_timeout_seconds and task_watch_poll_interval
So if you're asking generally “how long will a worker observe a flow run?”:
- Kubernetes worker: job_watch_timeout_seconds
- Most other workers in 2.20: no direct equivalent
If helpful, I can also show you where to set job_watch_timeout_seconds in a Kubernetes work pool / deployment for Prefect 2.20.Ferchau Gupta
05/28/2026, 6:47 AMMarvin
05/28/2026, 6:47 AMMarvin
05/28/2026, 6:51 AMtimeout_seconds on a flow is a Prefect engine timeout, not a Kubernetes job kill timeout.
So the first thing to debug is: is the flow timing out in Prefect state, but the k8s pod/job keeps running?
Those are two different layers.
What timeout_seconds does in Prefect 2.20
- It tells the Prefect engine to mark the flow run as timed out
- It does not automatically set a Kubernetes activeDeadlineSeconds
- It does not guarantee the underlying pod/job is forcibly terminated
- For long blocking sync code, timeout handling may only happen at interruptible boundaries
So you can end up with:
- flow run state in Prefect = TimedOut / failed
- k8s pod/job = still running
How to debug it
1. Confirm the Prefect flow run state
Check whether Prefect thinks the run timed out:
bash
prefect flow-run inspect <FLOW_RUN_ID>
Look for:
- state type FAILED
- state name like TimedOut
- message similar to Flow run exceeded timeout of X seconds
If the run is not entering a timed out state at all, that points to flow execution behavior.
If it is entering timed out state, but pod keeps running, that points to Kubernetes/infrastructure cleanup.
2. Check the flow run logs
Follow logs for the run:
bash
prefect flow-run logs <FLOW_RUN_ID>
You want to see whether there is a timeout-related message around the expected time.
If you can, also set:
bash
PREFECT_LOG_LEVEL=DEBUG
in the job/pod environment and rerun. Debug logs can show more about engine behavior.
3. Inspect the Kubernetes job/pod directly
After the timeout should have happened:
bash
kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
Questions to answer:
- Is the pod still Running?
- Did the Python process inside the pod actually exit?
- Is the pod stuck because some subprocess is still alive?
- Is there any traceback or timeout log in container logs?
If the Prefect flow state timed out but the pod is still running, the issue is almost certainly that nothing is telling Kubernetes to kill the pod.
4. Check whether your flow code is interruptible
This is a big one in Prefect 2.x.
A flow timeout is not always a hard immediate kill. If your flow does long blocking work like:
python
@flow(timeout_seconds=60)
def my_flow():
time.sleep(3600)
the timeout may not interrupt that cleanly right away.
Similarly, if your flow is spending a long time in:
- blocking I/O
- external subprocesses
- non-cooperative synchronous loops
- libraries that don’t yield control
then the timeout can appear ineffective.
A good debug test is to compare these two cases.
Potentially problematic:
python
from prefect import flow
import time
@flow(timeout_seconds=10)
def blocked_flow():
time.sleep(300)
More cooperative:
python
from prefect import flow, task
import time
@task
def tick():
return
@flow(timeout_seconds=10)
def cooperative_flow():
for _ in range(300):
time.sleep(1)
tick()
If the second behaves better, the issue is likely interruptibility of your code path.
5. Distinguish flow timeout from worker/job watch timeout
Earlier you asked about job_watch_timeout_seconds.
That setting is for the worker’s monitoring of the k8s job, not the flow’s own runtime timeout.
So:
- flow(timeout_seconds=...) = Prefect flow execution timeout
- job_watch_timeout_seconds = how long the worker watches the k8s job/events
These do not enforce the same thing.
6. Check your k8s job manifest for a hard runtime limit
If you need the pod/job to actually stop, inspect whether your job spec has:
yaml
spec:
activeDeadlineSeconds: 70
Without that, Kubernetes itself has no hard stop tied to your flow timeout.
That’s the most practical workaround in Prefect 2.20:
- keep timeout_seconds for Prefect state behaviorMarvin
05/28/2026, 6:51 AMactiveDeadlineSeconds in the k8s job so the pod is actually killed
Usually you’d set activeDeadlineSeconds slightly larger than the flow timeout.
Example:
- flow timeout_seconds=600
- k8s activeDeadlineSeconds=660
7. Check whether retries are masking what you see
If your flow has retries configured, a timeout may transition into a retry path instead of looking like a final failed run immediately.
So inspect:
- retries
- retry_delay_seconds
If retries are on, that can make timeout behavior look confusing.
8. Verify whether you are using old Kubernetes infrastructure vs worker-based execution
In Prefect 2.20, some users are still on older KubernetesJob infrastructure patterns, others use workers/work pools.
That matters because debugging points differ a bit:
- old infrastructure block/job spec customization
- worker/work pool base job template customization
If you want, I can help with either, but I’d need to know which one you’re using.
What I would do first
1. Run a tiny repro flow with timeout_seconds=30
2. Make it intentionally block
3. Inspect:
- Prefect state
- pod status
- pod logs
4. Confirm whether:
- neither Prefect nor k8s stops it
- Prefect times out but k8s pod remains alive
That tells us exactly where the failure is.
Recommended practical fix in Prefect 2.20
For k8s-backed flow runs, use both:
- Prefect timeout_seconds on the flow
- Kubernetes activeDeadlineSeconds on the job/pod
Because in 2.20, Prefect timeout alone is not a reliable infrastructure-level kill mechanism.
If you share:
- whether you use KubernetesJob or kubernetes work pool/worker
- a sample flow
- your deployment/work pool job template
I can help you pinpoint the exact place to add the hard timeout and what logs to inspect.Ferchau Gupta
05/29/2026, 8:36 AMMarvin
05/29/2026, 8:36 AMMarvin
05/29/2026, 8:39 AMtimeout_seconds does not reliably clean up the underlying k8s job/pod
- Early Prefect 3: still not fully solved
- Prefect 3.7.1+: Yes, substantially improved — flow timeout can trigger infrastructure teardown, including Kubernetes job deletion
So if you mean “does a timed-out flow now also stop the k8s infrastructure?” the answer is:
In modern Prefect 3, yes — this is handled much better than in 2.20.
What changed is that when a flow times out, Prefect can now invoke worker-side infrastructure cleanup via the worker’s kill_infrastructure() implementation. The Kubernetes worker supports that and deletes the job/pods.
Important caveats
- This is not the same as Kubernetes-native timeout enforcement like activeDeadlineSeconds
- Prefect timeout is still initiated at the Prefect/runtime level first
- Kubernetes cleanup then happens as a teardown action
- For hard infrastructure guarantees, activeDeadlineSeconds is still a good safety net
So the practical recommendation in Prefect 3 is:
1. Use Prefect 3.7.1+ if possible
2. Set flow timeout_seconds
3. For Kubernetes, optionally also set activeDeadlineSeconds as defense in depth
That gives you:
- Prefect-level timeout semantics
- worker-driven cleanup
- optional Kubernetes-native hard stop
One more caveat:
If your code is stuck in certain blocking sync operations, timeout behavior can still be less reliable than fully async/cooperative code. So upgrading helps a lot, but it doesn’t make every blocking Python call instantly interruptible.
If you want, I can show you:
- how to configure this in a Prefect 3 Kubernetes work pool, or
- how to check whether your current Prefect 3 version includes the timeout cleanup behavior.