<@ULVA73B9P> How can I check which database is the...
# marvin-ai
f
@Marvin How can I check which database is the prefect server running with if I started it without specifying any
PREFECT__SERVER__DATABASE__CONNECTION_URL
? I am using Prefect 2.20
m
thought for 91.4 seconds
In Prefect 2.20, if you didn't set a database connection URL, Prefect Server defaults to SQLite. By default, the DB is typically at:
Copy code
bash
~/.prefect/prefect.db
There’s also some backward-compatibility logic: if
prefect.db
does not exist but
~/.prefect/orion.db
does, Prefect may use
orion.db
instead. Best ways to check 1. Show the active config, including defaults
Copy code
bash
prefect config view --show-defaults --show-sources
Look for:
Copy code
bash
PREFECT_API_DATABASE_CONNECTION_URL='sqlite+aiosqlite:///.../.prefect/prefect.db'
PREFECT_HOME='.../.prefect'
2. Check whether the DB files exist
Copy code
bash
ls -lah ~/.prefect/prefect.db
ls -lah ~/.prefect/orion.db
3. Inspect it in Python
Copy code
python
from prefect.settings import PREFECT_API_DATABASE_CONNECTION_URL, PREFECT_HOME

print("DB URL:", PREFECT_API_DATABASE_CONNECTION_URL.value())
print("PREFECT_HOME:", PREFECT_HOME.value())
That should print the resolved connection URL Prefect is using. Important note on the env var name for Prefect 2.x In Prefect 2.x, the setting you usually want to inspect is:
Copy code
bash
PREFECT_API_DATABASE_CONNECTION_URL
not
PREFECT__SERVER__DATABASE__CONNECTION_URL
. So you can also check directly:
Copy code
bash
echo $PREFECT_API_DATABASE_CONNECTION_URL
env | grep ^PREFECT_
If you want, I can also show you how to check this from a running server process or help confirm whether your instance is using SQLite vs Postgres.
f
@Marvin Is there any variable like
job_watch_timeout_seconds
in Prefect 2.20, which defines the number of seconds for which a worker will observe a flow run?
m
thought for 158.5 seconds
Yes — but in Prefect 2.20,
job_watch_timeout_seconds
is not a global worker setting. It exists for Kubernetes workers / Kubernetes job infrastructure, and it controls how long the worker will keep watching the Kubernetes job before timing out. For Kubernetes in Prefect 2.20 - `job_watch_timeout_seconds`: max time to watch the job overall - `pod_watch_timeout_seconds`: max time to wait for pod creation If
job_watch_timeout_seconds
is
None
(the default), the worker waits indefinitely. Important: this is not a setting for all worker types. Other worker types in 2.20 - Process worker: no equivalent watch-timeout setting - Docker worker: no equivalent watch-timeout setting - Vertex AI worker: has
job_watch_poll_interval
, but that is polling frequency, not a total timeout - AWS ECS: uses different settings such as
task_start_timeout_seconds
and
task_watch_poll_interval
So if you're asking generally “how long will a worker observe a flow run?”: - Kubernetes worker:
job_watch_timeout_seconds
- Most other workers in 2.20: no direct equivalent If helpful, I can also show you where to set
job_watch_timeout_seconds
in a Kubernetes work pool / deployment
for Prefect 2.20.
f
@Marvin in Prefect 2.20, when running flow runs on k8s infrastructure, setting the timeout_seconds on flow level is not working. The flow keeps running even after the stipulated timeout. How can I debug the issue behind?
m
thought for 232.9 seconds
What you’re seeing is consistent with a known limitation/pattern in Prefect 2.20 on Kubernetes:
timeout_seconds
on a flow is a Prefect engine timeout, not a Kubernetes job kill timeout.
So the first thing to debug is: is the flow timing out in Prefect state, but the k8s pod/job keeps running? Those are two different layers. What
timeout_seconds
does in Prefect 2.20
- It tells the Prefect engine to mark the flow run as timed out - It does not automatically set a Kubernetes
activeDeadlineSeconds
- It does not guarantee the underlying pod/job is forcibly terminated - For long blocking sync code, timeout handling may only happen at interruptible boundaries So you can end up with: - flow run state in Prefect =
TimedOut
/ failed - k8s pod/job = still running How to debug it 1. Confirm the Prefect flow run state Check whether Prefect thinks the run timed out:
Copy code
bash
prefect flow-run inspect <FLOW_RUN_ID>
Look for: - state type
FAILED
- state name like
TimedOut
- message similar to
Flow run exceeded timeout of X seconds
If the run is not entering a timed out state at all, that points to flow execution behavior. If it is entering timed out state, but pod keeps running, that points to Kubernetes/infrastructure cleanup. 2. Check the flow run logs Follow logs for the run:
Copy code
bash
prefect flow-run logs <FLOW_RUN_ID>
You want to see whether there is a timeout-related message around the expected time. If you can, also set:
Copy code
bash
PREFECT_LOG_LEVEL=DEBUG
in the job/pod environment and rerun. Debug logs can show more about engine behavior. 3. Inspect the Kubernetes job/pod directly After the timeout should have happened:
Copy code
bash
kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>
Questions to answer: - Is the pod still
Running
? - Did the Python process inside the pod actually exit? - Is the pod stuck because some subprocess is still alive? - Is there any traceback or timeout log in container logs? If the Prefect flow state timed out but the pod is still running, the issue is almost certainly that nothing is telling Kubernetes to kill the pod. 4. Check whether your flow code is interruptible This is a big one in Prefect 2.x. A flow timeout is not always a hard immediate kill. If your flow does long blocking work like:
Copy code
python
@flow(timeout_seconds=60)
def my_flow():
    time.sleep(3600)
the timeout may not interrupt that cleanly right away. Similarly, if your flow is spending a long time in: - blocking I/O - external subprocesses - non-cooperative synchronous loops - libraries that don’t yield control then the timeout can appear ineffective. A good debug test is to compare these two cases. Potentially problematic:
Copy code
python
from prefect import flow
import time

@flow(timeout_seconds=10)
def blocked_flow():
    time.sleep(300)
More cooperative:
Copy code
python
from prefect import flow, task
import time

@task
def tick():
    return

@flow(timeout_seconds=10)
def cooperative_flow():
    for _ in range(300):
        time.sleep(1)
        tick()
If the second behaves better, the issue is likely interruptibility of your code path. 5. Distinguish flow timeout from worker/job watch timeout Earlier you asked about
job_watch_timeout_seconds
. That setting is for the worker’s monitoring of the k8s job, not the flow’s own runtime timeout. So: -
flow(timeout_seconds=...)
= Prefect flow execution timeout -
job_watch_timeout_seconds
= how long the worker watches the k8s job/events These do not enforce the same thing. 6. Check your k8s job manifest for a hard runtime limit If you need the pod/job to actually stop, inspect whether your job spec has:
Copy code
yaml
spec:
  activeDeadlineSeconds: 70
Without that, Kubernetes itself has no hard stop tied to your flow timeout. That’s the most practical workaround in Prefect 2.20: - keep
timeout_seconds
for Prefect state behavior
- also set
activeDeadlineSeconds
in the k8s job so the pod is actually killed Usually you’d set
activeDeadlineSeconds
slightly larger than the flow timeout. Example: - flow
timeout_seconds=600
- k8s
activeDeadlineSeconds=660
7. Check whether retries are masking what you see If your flow has retries configured, a timeout may transition into a retry path instead of looking like a final failed run immediately. So inspect: -
retries
-
retry_delay_seconds
If retries are on, that can make timeout behavior look confusing. 8. Verify whether you are using old Kubernetes infrastructure vs worker-based execution In Prefect 2.20, some users are still on older
KubernetesJob
infrastructure patterns, others use workers/work pools. That matters because debugging points differ a bit: - old infrastructure block/job spec customization - worker/work pool base job template customization If you want, I can help with either, but I’d need to know which one you’re using. What I would do first 1. Run a tiny repro flow with
timeout_seconds=30
2. Make it intentionally block 3. Inspect: - Prefect state - pod status - pod logs 4. Confirm whether: - neither Prefect nor k8s stops it - Prefect times out but k8s pod remains alive That tells us exactly where the failure is. Recommended practical fix in Prefect 2.20 For k8s-backed flow runs, use both: - Prefect
timeout_seconds
on the flow - Kubernetes
activeDeadlineSeconds
on the job/pod Because in 2.20, Prefect timeout alone is not a reliable infrastructure-level kill mechanism. If you share: - whether you use KubernetesJob or kubernetes work pool/worker - a sample flow - your deployment/work pool job template I can help you pinpoint the exact place to add the hard timeout and what logs to inspect.
f
@Marvin is this issue solved in Prefect 3?
m
thought for 194.0 seconds
Yes — but only in newer Prefect 3 releases. Short version: - Prefect 2.20: No, flow
timeout_seconds
does not reliably clean up the underlying k8s job/pod - Early Prefect 3: still not fully solved - Prefect 3.7.1+: Yes, substantially improved — flow timeout can trigger infrastructure teardown, including Kubernetes job deletion So if you mean “does a timed-out flow now also stop the k8s infrastructure?” the answer is: In modern Prefect 3, yes — this is handled much better than in 2.20. What changed is that when a flow times out, Prefect can now invoke worker-side infrastructure cleanup via the worker’s
kill_infrastructure()
implementation. The Kubernetes worker supports that and deletes the job/pods. Important caveats - This is not the same as Kubernetes-native timeout enforcement like
activeDeadlineSeconds
- Prefect timeout is still initiated at the Prefect/runtime level first - Kubernetes cleanup then happens as a teardown action - For hard infrastructure guarantees,
activeDeadlineSeconds
is still a good safety net So the practical recommendation in Prefect 3 is: 1. Use Prefect 3.7.1+ if possible 2. Set flow
timeout_seconds
3. For Kubernetes, optionally also set
activeDeadlineSeconds
as defense in depth That gives you: - Prefect-level timeout semantics - worker-driven cleanup - optional Kubernetes-native hard stop One more caveat: If your code is stuck in certain blocking sync operations, timeout behavior can still be less reliable than fully async/cooperative code. So upgrading helps a lot, but it doesn’t make every blocking Python call instantly interruptible. If you want, I can show you: - how to configure this in a Prefect 3 Kubernetes work pool, or - how to check whether your current Prefect 3 version includes the timeout cleanup behavior.