<@ULVA73B9P> I am encountering the issue that the ...
# ask-marvin
f
@Marvin I am encountering the issue that the tasks runs are missing in the Prefect run logs. The flow executes as expected but the just the logs for the task runs are not recorded. The same issue was raised https://github.com/PrefectHQ/prefect/issues/15153 and I followed the debug steps but no luck. I verified that the running container can talk to the postgress database. No task runs were recorded in the db when I use prefect client to check. Does anyone know the fix? Config: • Self-hosting Prefect API server + Prefect worker + Postgress db in the ECS. • The prefect version "3.7.20" (derived using
curl https://<prefect-server-url>/api/admin/version
)
m
thought for 491.9 seconds
Thanks for the detailed context — a few things can cause “flow runs appear but there are no task runs/logs” in Prefect 3.x, especially on self-hosted servers. Here’s how I’d triage it, plus a couple of likely culprits and fixes. First, some quick context and references: - The symptom matches a few known edge cases from recent versions, e.g. https://github.com/PrefectHQ/prefect/issues/15153 and a related decorator case in https://github.com/PrefectHQ/prefect/issues/16182. In both, tasks executed but the server never recorded task runs. - In Prefect 3.x, task runs are created from events published by your running flow. If those events don’t get published (e.g., client-side error, wrapper/decorator bypass, or version mismatch), you’ll see exactly what you’re describing. High-signal checks (in order) 1) Make sure your task calls are actually creating task runs - If you use
.fn
(e.g.,
my_task.fn(...)
) or call the underlying function via a wrapper, Prefect does not create a task run — it just executes Python code. This is the most common cause of “flows run but no task runs.” - Also check for additional decorators around your tasks/flows. In Prefect 3.x, the Prefect decorator generally needs to be the outermost decorator. Wrapping a
@task
or
@flow
with your own decorator can strip the Prefect instrumentation and lead to an empty graph and missing logs (very similar to https://github.com/PrefectHQ/prefect/issues/16182). - Good:
Copy code
@task(log_prints=True)
    def do_thing(): ...

    @flow
    def my_flow():
        do_thing()         # or do_thing.submit()
- Risky:
Copy code
@my_decorator
    @task
    def do_thing(): ...

    @my_decorator
    @flow
    def my_flow(): ...
- If you must decorate, ensure
@task
/
@flow
is outermost and your decorator uses
functools.wraps
properly. 2) Verify Prefect versions in ALL containers match (server, worker, and the image that runs your flows) - A version mismatch can break event publishing even if runs otherwise “work”. - Please run
prefect version
inside: - the API server container - the worker container - the flow execution container/image in ECS - Make sure they’re all on the same 3.7.20 or, preferably, upgrade all of them to the same newer patch/minor. If you can, try the latest 3.8.x+ across the board; there have been numerous event/logging robustness fixes since early 3.x. 3) Check worker/execution logs for event publishing failures - Look for errors like: - “EventsWorker failed with X pending items” - “Could not publish event” / httpx errors - If you see these, the flow engine tried to create task runs, but event publishing failed. This can be caused by: - transient networking through an ALB/ingress - TLS/HTTP2 oddities - client/server version mismatch - To get more detail, set
PREFECT_LOGGING_LEVEL=DEBUG
in the worker/execution environment and re-run a small flow. 4) Minimal, known-good test on your stack - Deploy/run this minimal flow unchanged on the same infrastructure:
Copy code
from prefect import flow, task

  @task(log_prints=True)
  def t(name):
      print(f"hello from {name}")

  @flow(log_prints=True)
  def demo():
      t("call")        # creates a task run
      t.submit("sub")  # also creates a task run

  if __name__ == "__main__":
      demo()
- You should see two task runs under the flow run. If you do not, it’s almost certainly an event publishing / instrumentation / version issue rather than your business logic. 5) Sanity-check API URL and server logs - Since your flow runs are appearing,
PREFECT_API_URL
is likely correct. Still confirm the exact value inside the execution container matches the server you queried with curl. - On the API server, check logs around the time of your runs for “event ingestion” or “graph” errors. If none of the above resolve it, could you share the following so we can pinpoint this quickly?
- A small code snippet of a representative task + flow, including any decorators you’re using - The exact Prefect versions from server, worker, and the execution image (
prefect version
from each) - Any related worker/execution logs, especially mentions of “EventsWorker” or httpx errors - Whether your tasks are ever called via
.fn
or wrapped by another decorator Why I suspect instrumentation or versioning here - Your description matches two known patterns: tasks running as plain Python (due to
.fn
or wrapping decorators) and event publishing failures (e.g., “EventsWorker failed…” in #15153). - You’re on a later 3.7.x server, but the execution image may not be. When the client and server are out of sync, the event pathway that creates task runs can break even if the flow succeeds. References - Issue about missing task runs in 3.x: https://github.com/PrefectHQ/prefect/issues/15153 - Decorator-related empty graph/no task runs: https://github.com/PrefectHQ/prefect/issues/16182 - Settings and profiles: https://docs-3.prefect.io/v3/concepts/settings-and-profiles - Server concepts: https://docs-3.prefect.io/v3/concepts/server If you can share the snippets and versions, I’m happy to help you zero in on the exact cause.