https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have on-prem Prefect open source wi...
# ask-marvin
r
@Marvin I have on-prem Prefect open source with process workers. Let's discuss: I have some running flow. In same time all workers are intentionally restarted. I observed, that flow stayed in running state. What would you recommend, how to manage to automatically return this flow in order to finish all work ?
m
✅ thought for 151.5 seconds
Yes — with process workers in Prefect OSS, what you observed is expected. If a worker is restarted while it is executing a flow run, the server may still show that run as Running, because the worker that would normally report the final state is gone. In OSS, there is no built-in server-side orphaned-run recovery that automatically says “this running flow lost its worker, put it back in a schedulable state and continue.” Important consequence: a flow run that was already started generally cannot be resumed from the middle. The usual recovery model is: 1. detect that the run is orphaned 2. mark it 
Crashed
or 
Failed
3. rely on flow retries or external logic to start it again 4. make the flow itself idempotent / checkpointed so reruns safely continue work Recommendation 1. Design for rerun, not in-place resume Prefect does not resume an interrupted Python process from its exact execution point. So the robust pattern is: - make tasks/flow steps idempotent - persist progress externally - on rerun, skip already-finished work Examples: - write processed item IDs to a database table - write checkpoints/results to object storage - upsert instead of insert - make downstream side effects safe to repeat 2. Use retries on the flow run If the run is eventually marked `Crashed`/`Failed`, retries can restart it automatically. Example:
Copy code
python
from prefect import flow

@flow(retries=3, retry_delay_seconds=60)
def my_flow():
    ...
But note: retries only help after the run reaches a retryable terminal state. If it stays stuck in 
Running
, something must transition it first. 3. Add external cleanup for orphaned running flows In OSS, the practical solution is to run a watchdog script or cron job that: - finds flow runs stuck in 
Running
for too long - checks whether the responsible worker is gone - marks the run 
Crashed
- lets retries or automation re-execute it That is the main operational pattern for self-hosted/process-worker deployments. Conceptually:
Copy code
python
from prefect import get_client
from prefect.states import Crashed
import asyncio

async def cleanup_orphaned_runs():
    async with get_client() as client:
        # find runs that have been Running longer than expected
        # then set state to Crashed
        await client.set_flow_run_state(
            flow_run_id=<run_id>,
            state=Crashed(message="Worker restarted; run orphaned"),
            force=True,
        )

asyncio.run(cleanup_orphaned_runs())
If you want, I can help you build a complete watchdog script for Prefect 3. 4. Prefer graceful worker shutdowns when possible If you intentionally restart workers, try to avoid killing active runs abruptly. A graceful restart strategy is much safer: - stop sending new work to workers - wait for active flow runs to finish - then restart workers This is especially important for 
process
workers. 5. Run multiple workers, but know the limitation Multiple workers improve availability for new runs, but they do not rescue a flow process that was already running on a worker that died. That specific run still needs cleanup + retry/restart. 6. Put timeout boundaries on flows Timeouts help prevent “forever running” behavior from the flow logic side:
Copy code
python
from prefect import flow

@flow(timeout_seconds=3600, retries=2, retry_delay_seconds=120)
def my_flow():
    ...
This is useful, but if the worker disappears entirely, the timeout may not be enough by itself to clean up the server-side state promptly. What I would do in your setup For on-prem OSS + process workers, I’d recommend this combination: - Make flows idempotent - Enable flow retries - Use graceful restarts whenever possible - Run a watchdog job that marks stale 
Running
runs as 
Crashed
- Optionally alert when workers go offline or when runs exceed expected duration That gives you reliable automatic recovery even though true mid-process resume is not available.
Why the run stays Running Because the worker owns the local subprocess and usually reports the terminal state back to the API. If the worker disappears first, the API may never receive that update, so the run remains stuck in 
Running
. Relevant CLI To run workers:
Copy code
bash
prefect worker start --pool <pool-name>
Verified via CLI help. Docs / links - Prefect docs: Prefect documentation - Work pools and workers: Work pools overview - Flow retries: Flow retries If you want, I can provide one of these next: 1. a watchdog script to auto-crash stale running runs in OSS 2. a recommended idempotent flow pattern 3. a graceful restart procedure for process workers
r
@Marvin My questions: 1. Is acceptable to use task's cache as checkpoint technique, assuming that task's body is idempotent ? 2. Is flow retryable if flow's process started by workek is gone / killed ? 3. Do you recommend to run watchdog as ordinary Prefect deployed flow ? If yes what scheduling params would you recommend ?
m
✅ thought for 134.8 seconds
Good questions. Short answers: 1. Yes, with limits — task cache can be used as a lightweight checkpoint mechanism if the task is idempotent and its result is persisted somewhere durable/shared. 2. Not automatically in the situation you described — if the worker/process disappears and the flow run stays 
Running
, retries will not fire until something moves it to a retryable/terminal state. 3. Yes, a watchdog as a deployed Prefect flow is reasonable in OSS. I’d usually schedule it every 5–10 minutes. A bit more detail on each: --- 1) Is task cache acceptable as a checkpoint technique? Yes, but think of it as “recompute avoidance”, not true resume. It works well when: - the task is idempotent - the task result is persisted - the cache storage is durable and accessible after worker restart - rerunning the flow can safely skip already-completed tasks Example shape:
Copy code
python
from prefect import task, flow
from prefect.cache_policies import INPUTS

@task(
    cache_policy=INPUTS,
    persist_result=True,
)
def step(x):
    # safe to repeat
    return x * 2

@flow
def my_flow():
    a = step(1)
    b = step(a)
    return b
What this gives you If the flow is rerun, Prefect may reuse the completed task state/result instead of executing the task again. What it does not give you - no resume from the middle of Python execution inside a task - no recovery of partially completed task body - no guarantee if result storage is local-only and your worker was restarted elsewhere Main caveats - If your result storage is local filesystem on the worker host, cache may be fragile depending on how you restart/run workers. - If a task has side effects, cache only works safely when those side effects are idempotent. - If a task dies halfway through, no usable cache entry is produced for that attempt. My recommendation Use task cache as a checkpoint-like optimization for: - expensive reads - pure transforms - deterministic chunk processing For stronger checkpointing, persist progress in an external system: - DB table of completed chunks - object storage manifest - “last processed offset” record - upserted outputs That pattern is more reliable than cache alone. --- 2) Is flow retryable if the flow process started by worker is gone/killed? Only after the flow run is transitioned out of stuck 
Running
. That distinction matters. If the worker is alive long enough to observe the subprocess crash and report a terminal state, then recovery is possible. If the worker itself disappears first: - the run may remain 
Running
- no retry happens while it is still 
Running
So in your scenario: - worker killed/restarted abruptly - flow run remains Running - => flow retry does not automatically start You need a cleanup step: - mark stale run as 
Crashed
or 
Failed
- then re-execute / retry it Practical takeaway Flow retries are useful, but in OSS they are not sufficient by themselves for orphaned runs caused by worker loss. So the recovery chain is usually:
Copy code
text
worker dies
-> flow run stuck in Running
-> watchdog marks run Crashed
-> external logic / rerun / retry policy starts a new attempt
If you want fully automatic behavior, your watchdog should not only detect/crash stale runs, but also optionally trigger a rerun strategy. --- 3) Should watchdog be an ordinary deployed Prefect flow? Yes, that is a reasonable pattern for self-hosted OSS. It’s nice because: - visible in the UI - scheduled normally - easy to version and test - can emit alerts/events/logs - can reuse Prefect API client Scheduling recommendation I’d start with: - interval: every 
5 minutes
- stale threshold: 
10–15 minutes
Why not the same value? Because you want to avoid false positives from: - temporary API hiccups - short worker restarts - clock skew / delayed scheduling A good starting policy: - watchdog runs every 
5 min
- marks runs suspicious only if 
Running
longer than expected and older than 
10–15 min
beyond normal heartbeat tolerance If your flows are short-lived and frequent, you can tighten it: - run every 
2–3 min
- stale threshold 
5–8 min
If your flows are long-running: - run every 
10 min
- stale threshold 
15–30 min
Important deployment advice Do not run the watchdog only on the same fragile worker group it is protecting, if possible. Better: - separate worker/pool for ops flows - or at least a distinct worker process - so the watchdog still runs when app workers are being restarted That’s probably the single biggest practical recommendation here. --- My preferred design I’d use all of these together: For business flows - idempotent tasks - persistent task results where useful - external progress markers for chunk-level completion - flow/task retries For operations - watchdog deployment on separate worker/pool - schedule every 
5 min
- stale threshold 
10–15 min
- mark orphaned runs 
Crashed
- optionally trigger rerun of eligible deployments --- Rule of thumb on cache vs checkpoint Use cache when: - “If this finished once, I can safely reuse its output.” Use external checkpoint/progress state when: - “I must know exactly which units of work already succeeded.” For production recovery, the second is more robust. --- Suggested starting parameters I’d start with:
Copy code
text
watchdog schedule: every 5 minutes
orphan threshold: 15 minutes
lookback window: last 24 hours
watchdog retries: 2-3
watchdog on separate work pool: yes
And for your flows:
Copy code
python
@flow(retries=2, retry_delay_seconds=60)
def my_flow():
    ...
plus task-level persisted caching where it really helps. --- Links - Prefect docs: Prefect documentation - Work pools / workers: Work pools overview - Retries: Flow retries - Caching concepts: Caching and results If you want, I can next write a concrete Prefect 3 watchdog flow that: - finds stale 
Running
runs - marks them 
Crashed
- and optionally triggers a rerun policy safely.
r
@Marvin Let's discuss watchdog. 1. How to make watch dog resilient against same situlation : worker running watchdog
m
✅ thought for 16.2 seconds
The key is: don’t let the watchdog depend on the exact same failure domain it is supposed to heal. If your watchdog runs on the same process-worker fleet, and you restart that fleet, then the watchdog dies too and cannot clean up orphaned runs. Recommended pattern 1. Run the watchdog on separate infrastructure Best option: - create a dedicated work pool for operational flows - run at least one separate worker for that pool - avoid restarting it together with your application workers So you have something like:
Copy code
text
pool: app-process-pool
  workers: worker-a, worker-b, worker-c

pool: ops-watchdog-pool
  workers: watchdog-worker-1
Then deploy the watchdog flow only to 
ops-watchdog-pool
. This is the cleanest approach. --- 2. Make the watchdog itself stateless and idempotent The watchdog should be safe to rerun at any time. That means: - query current 
Running
runs - decide whether they are stale - attempt state transition only if still stale - tolerate races if a run finishes while being inspected So if watchdog run N dies midway, run N+1 can do the same work safely. --- 3. Schedule it frequently enough that missing one run is fine Do not depend on a single execution. Example: - schedule every 
5 minutes
- stale threshold 
15 minutes
Then if one watchdog run is missed because of restart, the next one still catches orphaned runs. This gives resilience by redundancy in time. --- 4. Add retries to watchdog tasks/flow This protects against transient API/database issues, though not total worker loss. Example:
Copy code
python
from prefect import flow, task

@task(retries=3, retry_delay_seconds=10)
def find_orphans():
    ...

@task(retries=3, retry_delay_seconds=10)
def crash_orphans():
    ...

@flow(retries=2, retry_delay_seconds=60)
def watchdog():
    ...
--- 5. Use concurrency control so only one watchdog instance acts at a time You generally do not want multiple watchdog runs racing each other. Use a deployment schedule that is simple, and ideally ensure one active watchdog at a time. If your environment may overlap executions, add a guard such as: - a Prefect concurrency limit - or an external lock in DB/Redis/file That way: - only one watchdog performs cleanup - retries/next scheduled run do not double-act unnecessarily --- 6. Separate “detect” from “repair” logic mentally A robust watchdog often follows this pattern: - detect candidate orphaned runs - re-check candidate just before repair - repair only if still 
Running
and still stale That second check is important because a run may complete naturally while the watchdog is working. --- 7. Prefer broad thresholds over aggressive ones If your watchdog is too aggressive, it may mark valid long-running jobs as crashed. So for resilience: - heartbeat/restart turbulence should not trigger false positives - choose stale threshold comfortably above normal worker restart time For example: - if worker restarts usually finish within 2–3 minutes - use threshold like 10–15 minutes --- 8. Optionally run the watchdog outside Prefect If you want maximum independence, the watchdog does not have to be a Prefect flow. You can run it as: - systemd timer - cron job - Kubernetes CronJob - external service script This is actually more resilient than a Prefect deployment, because it does not depend on Prefect workers to execute. So the tradeoff is: Watchdog as Prefect flow - easier observability in Prefect - simpler to manage in one platform - but still depends on worker infrastructure Watchdog as external cron/service - most resilient - independent of worker fleet - but less integrated into Prefect UI For your use case, if worker restarts are a known operational event, I would seriously consider an external watchdog. --- My recommendation for your case If you want strong resilience: Best - watchdog runs outside Prefect as cron/systemd/K8s CronJob - it uses Prefect API to inspect and repair stale runs
Good - watchdog is a Prefect deployment - but on a separate dedicated work pool/worker that is not restarted with app workers Not recommended - watchdog deployed to the same work pool/workers as normal flows --- Practical architecture
Copy code
text
Prefect server
  |
  |-- app pool
  |     |-- process worker 1
  |     |-- process worker 2
  |
  |-- ops pool
        |-- watchdog worker
Schedule: - watchdog every 
5 min
- stale threshold 
15 min
Behavior: - detect stale 
Running
- mark 
Crashed
- optionally trigger rerun / rely on recovery flow --- One subtle point Even the watchdog worker itself can die. That’s okay if: - it is scheduled repeatedly - it is idempotent - another future run can perform the same cleanup So resilience comes less from “watchdog never fails” and more from: - independent placement - repeat scheduling - idempotent repair logic If you want, I can next propose: 1. external watchdog script approach, or 2. Prefect-deployed watchdog architecture with exact flow logic.
r
@Marvin For now I would make watchdog as prefect flow. How to schedule it, to achieve: 1. Late runs are not started (they are not needed) 2. Only one instance is started every 5 minutes ?
m
✅ thought for 139.2 seconds
Yes — for a watchdog flow, the usual goal is: - run it regularly - but if a scheduled run is late/stale, don’t bother starting it later That is a very sensible requirement. For Prefect 3 OSS, I’d recommend this pattern: Recommended setup 1. schedule watchdog on a short interval, like every 
5 minutes
2. make watchdog logic self-skip if it is too late 3. keep watchdog concurrency at 
1
4. keep the flow short and idempotent Why self-skip? Because in practice, the cleanest way to enforce “late runs are useless” is for the watchdog to inspect its own scheduled time / lateness at startup and immediately exit if too old. --- Recommended schedule For example: - interval: every 
5 minutes
- concurrency: 
1
- “acceptable lateness”: 
2–3 minutes
Meaning: - if the 10:00 watchdog run starts at 10:01, fine - if it starts at 10:08, it is obsolete, so exit immediately That avoids backlog processing. --- Why this is better than trying to rely only on server scheduling behavior Prefect has 
Late
state handling, but that is not the same as “never start stale run in all situations.” A run may still exist and eventually get picked up depending on timing and queue/worker conditions. For a watchdog, the safest pattern is: - schedule normally - reject obsolete execution in flow code This gives exact control. --- Concrete pattern
Copy code
python
from datetime import datetime, timezone, timedelta
from prefect import flow, runtime

MAX_ALLOWED_LATENESS = timedelta(minutes=3)

@flow(
    name="orphan-watchdog",
    log_prints=True,
)
def orphan_watchdog():
    scheduled_start = runtime.flow_run.scheduled_start_time

    # Ad hoc/manual run: no scheduled time, allow execution
    if scheduled_start is not None:
        now = datetime.now(timezone.utc)
        lateness = now - scheduled_start

        if lateness > MAX_ALLOWED_LATENESS:
            print(
                f"Skipping stale watchdog run. "
                f"Scheduled for {scheduled_start}, lateness={lateness}."
            )
            return

    print("Running watchdog logic...")
    # detect orphaned flow runs
    # mark stale ones crashed
    # optionally trigger recovery
This is usually the pattern I’d trust most. --- Add concurrency limit = 1 You also want to avoid multiple watchdog runs overlapping. So for the deployment: - set concurrency limit to 
1
Then if one watchdog run is still active when the next one is scheduled, you avoid piling up multiple concurrent watchdogs. If you want, I can help with the exact deployment syntax for your setup. --- How I’d configure it in practice
Copy code
text
schedule interval: 5 minutes
deployment concurrency limit: 1
self-skip lateness threshold: 3 minutes
watchdog stale-flow threshold: 15 minutes
That means: - watchdog itself must start close to schedule - watchdog checks application runs only when they are stale much longer Those are two separate thresholds. --- Behavior you get If workers are down briefly: - one or more watchdog runs may be delayed - when one finally starts, it sees “I am too late” - it exits immediately - next timely run handles the real work That prevents old watchdog invocations from being meaningful after recovery. --- Alternative You could make the watchdog run with a Cron schedule instead of interval if you prefer exact wall-clock times, e.g. every 5 minutes:
Copy code
text
*/5 * * * *
But the anti-backlog mechanism should still be the same: check lateness at startup and skip if stale. --- My recommendation For your first version, do this: - deploy watchdog every 
5 min
- set deployment concurrency to 
1
- in flow code, skip if 
now - scheduled_start_time > 3 min
That is simple and robust. Links - Schedules: Prefect schedules - Work pools / workers: Work pools overview
- Runtime context: Runtime context If you want, I can next provide: 1. a complete watchdog flow example with this self-skip logic, or 2. a deployment example showing schedule + concurrency settings.
r
@Marvin Last question : is it possible to detect worker's id from orphaned run and check if worker is alive ? I am trying do not rely only on stale-flow threshold parameter ...
m
✅ thought for 222.0 seconds
Yes — partially, but not as a simple first-class 
flow_run.worker_id
field. Short version: - You can reliably know the work pool / work queue of a flow run - You may be able to infer the worker from emitted events - There is not a straightforward built-in persistent field on the flow run like 
worker_id
/ 
worker_name
that you can always read back later So: possible in some cases, but not something I would make the only decision criterion for recovery logic. --- What is available on the flow run A flow run can give you things like: - work pool / work queue association - 
infrastructure_pid
for process-worker execution But that is not the same as a durable “this exact worker owns me” reference. For process workers, 
infrastructure_pid
can be useful, but only if: - the watchdog can reason about the same machine/process namespace - and the PID is still meaningful there That usually breaks down if your watchdog is not colocated with that worker host. --- Worker identity: not a simple stored field As far as Prefect 3 OSS behavior goes, there is not a normal direct API field on the flow run like:
Copy code
text
flow_run.worker_id
flow_run.worker_name
that you can depend on for watchdog logic. So if your desired algorithm is:
Copy code
text
find orphaned run
-> read exact worker id from run
-> ping worker
-> decide
that is generally not available in a clean built-in way. --- What you can do instead Option 1: correlate via worker events A worker emits events around flow-run execution, and those can be used to infer which worker picked up the run. Conceptually: - query events related to the flow run - inspect whether there is a worker-related event - extract worker identity from that event - then query current workers in the pool and compare heartbeat / status This is the best “native Prefect” correlation approach if you want worker-awareness. But: - event history may not be the simplest thing to depend on for every watchdog pass - it is still more indirect than a dedicated ownership field --- Option 2: use worker liveness at the pool level Instead of proving “worker X for run Y is dead”, use a practical heuristic: - run is 
Running
- run is older than stale threshold - all workers in the relevant pool/queue are offline or absent That is much easier and often operationally good enough. This avoids needing exact run-to-worker mapping. --- Option 3: add your own attribution metadata If exact correlation matters a lot, a pragmatic pattern is to store additional metadata yourself at runtime. For example, the worker sets environment variables during execution, including worker attribution information. A flow can capture that early and persist it somewhere external. For example:
Copy code
python
import os
from prefect import flow

@flow
def my_flow():
    worker_name = os.getenv("PREFECT__WORKER_NAME")
    worker_id = os.getenv("PREFECT__WORKER_ID")

    # persist this mapping externally if needed
    print(worker_name, worker_id)
Then your watchdog can consult your own mapping store. That is often the most deterministic solution if you truly need exact worker identity. --- Would I recommend relying on worker identity for orphan cleanup? Only as a secondary signal. My recommendation is: Primary signal - run has been 
Running
too long Secondary signals - responsible worker appears offline, if inferable - no recent worker heartbeats in that pool - process PID no longer exists, if locally checkable - optional custom metadata says worker was host X / worker Y So the decision becomes: - stale enough - and supporting evidence suggests execution is gone That is much safer than stale-threshold-only, while still not requiring perfect worker mapping. --- Recommended watchdog decision policy For example: Crash a run only if: 1. it has been 
Running
longer than 
orphan_threshold
2. and at least one of these is true: - mapped worker is offline
- no workers in its pool are alive - infrastructure PID is gone - no custom heartbeat/progress marker updated recently That gives you a more conservative repair policy. --- My practical recommendation for your case Since you are using process workers on-prem OSS, I would use this layered approach: 1. Base threshold 
Running
longer than expected + margin 2. Worker/pool liveness check inspect workers in the work pool and their heartbeat timestamps 3. Optional event correlation try to infer the worker from worker-related events for the run 4. Optional custom progress heartbeat have the flow/task update an external progress timestamp If 2–4 indicate the run is abandoned, then mark it 
Crashed
. --- Bottom line Yes, partial worker detection is possible, but not via a simple guaranteed 
worker_id
field on the flow run. For a watchdog, I would treat worker liveness correlation as an enhancement, not the sole mechanism. Links - Work pools / workers: Work pools overview - Runtime context: Runtime context - Prefect docs: Prefect documentation If you want, I can next sketch a watchdog decision algorithm that combines: - stale threshold - pool worker heartbeats - optional worker-event correlation into one practical recovery rule.
r
@Marvin What about Prefect managed flow + process worker on prem. Are there any improvements regarding to our conversation ?
m
✅ thought for 155.3 seconds
Yes — Prefect Cloud + on-prem process workers does improve this situation in a meaningful way. The biggest improvement from our whole discussion is: Cloud can automatically detect unresponsive flow runs and mark them 
Crashed
. That is the main thing missing in self-hosted OSS by default. --- What improves with Prefect Cloud 1. Automatic unresponsive-run detection With Cloud, there is managed detection for flow runs that stop heartbeating. So if: - a process worker restarts - the flow process disappears - the run stops heartbeating Cloud can automatically detect that and transition the run out of stuck 
Running
. That materially improves your case because it removes the worst failure mode we discussed: - run remains 
Running
forever In Cloud, that can be handled automatically. --- 2. Less need for a custom watchdog just to unstick runs In OSS, a watchdog is often required just to: - detect stale 
Running
- mark it 
Crashed
In Cloud, that part is often already covered by managed automation / managed orchestration behavior for unresponsive runs. So your custom watchdog may become: - optional - simpler - focused on business-specific recovery policy instead of core liveness detection --- 3. Better event/automation capabilities Cloud gives you more orchestration-level features around: - automations - event-based actions - notifications - state-change handling So recovery flows can be cleaner than in OSS. For example, instead of polling aggressively, you may be able to rely more on platform-managed detection plus automations. --- What does NOT fundamentally change 1. Process workers still cannot resume a killed Python process in-place This remains the same. Even in Cloud: - if a running process is killed - execution is gone - Prefect does not restore the Python stack from the middle So the design principle is still: - make flows/tasks idempotent - use retries/reruns - persist progress externally if needed --- 2. Worker restart still interrupts local execution Cloud improves orchestration/recovery, not the fact that your process died. So: - the run may be auto-crashed - then retried/rerun - but not resumed in-memory --- 3. Exact flow-run-to-worker mapping is still not the main abstraction Cloud helps more with run liveness handling than with giving you a perfect “this worker owns this run forever” model. So even in Cloud, I would still think in terms of: - run heartbeat - worker heartbeat - automation on unresponsive runs not “recover by exact worker ownership.” --- How this changes my recommendation For OSS self-hosted I recommended: - dedicated watchdog flow or external watchdog - stale threshold logic - optional worker-liveness enrichment For Cloud + on-prem process workers I would recommend: 1. Enable/verify managed automations for unresponsive runs 2. Keep flows idempotent and retryable 3. Use task/result persistence where useful 4. Use a custom watchdog only for extra policy, not as the primary safety net Examples of “extra policy”: - auto-rerun only some deployments - alert if same deployment crashes repeatedly - distinguish “worker maintenance restart” from real failures - business-level cleanup --- Practical effect for your original scenario Scenario: - flow is running - all on-prem process workers are intentionally restarted In OSS: - run may stay 
Running
indefinitely unless your watchdog intervenes In Cloud: - the run should be detected as unresponsive after heartbeats stop - then it can be transitioned automatically to 
Crashed
- after that, your retry/rerun strategy can take over That is the biggest improvement. --- Would I still build a watchdog in Cloud? Maybe, but much smaller. I would not start with a custom “orphan detector” if managed unresponsive-run handling is enabled. I would consider a watchdog/recovery flow if you want custom behavior such as: - rerun only if deployment has label X
- suppress rerun during maintenance window - only rerun if crash count < N - route alerts differently by team So in Cloud, watchdog becomes policy automation, not basic liveness repair. --- Bottom line Yes, Prefect Cloud materially improves your situation for on-prem process workers, mainly because stuck running runs can be automatically detected and crashed when heartbeats stop. But it does not change the need for: - idempotent flows/tasks - rerun-based recovery instead of in-memory resume - careful restart strategy for workers Links - Prefect Cloud / docs: Prefect documentation - Automations: Automations - Work pools / workers: Work pools overview If you want, I can help you compare two concrete designs next: 1. OSS + custom watchdog 2. Cloud + managed unresponsive-run handling + minimal custom recovery flow
3 Views
Open in Slack
PreviousNext
Powered by