https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> We have prefect workers deployed in a...
# ask-marvin
a
@Marvin We have prefect workers deployed in aws ecs infrastructure. Tasks showed correctly in run DAGs in Prefect 3.4.10, but upgrading it to 3.7, tasks do not show up. Only subflow runs show. What do I need to fix?
m
✅ thought for 193.3 seconds
What changed is most likely not ECS itself, but how the UI DAG gets populated in newer 3.x versions. In Prefect 3.7, the DAG view only includes task runs that have actually been recorded with a non-
PENDING
state and a usable start timestamp. Subflow runs can still appear even when task runs do not, because they come from flow-run records and are queried differently. Most likely cause: your task runs are executing, but their task-run state/timestamps are not being recorded in a way the 3.7 DAG query accepts. What to check: 1. Inspect a flow run’s task runs in the DB / API For affected flow runs, confirm task runs have: - 
state_type
not equal to 
PENDING
- 
start_time
or 
expected_start_time
populated If task runs stay 
PENDING
, they will not appear in the DAG in 3.7. 2. Check worker / flow logs for task-run event or state-recording issues In particular, look for errors around: - task run state updates - event emission - API communication from the ECS task back to Prefect 3. If you use custom states/names, verify behavior after upgrade Prefect changed some event naming behavior in newer 3.7 releases, so anything depending on custom state/event names may no longer behave the same way. 4. Upgrade to the latest 3.7.x, not just 3.7.0 There were fixes in the 3.7 line related to event/state handling. If you’re on an early 3.7 release, upgrading further is a good first step. Why this happens: - The DAG view in 3.7 filters out task runs with 
PENDING
state. - It also requires a task run to have a start-related timestamp. - Subflows can still show because they’re represented as flow runs, not plain task runs. So the fix is usually: - make sure task runs are getting proper state transitions recorded - make sure the worker/flow process can report those states back to the API - update to latest 3.7.x Useful references: - Prefect releases: Prefect releases - Prefect 3 docs: Prefect docs If you want, I can help you narrow this down quickly if you share: - your exact Prefect version before/after - whether you use 
ConcurrentTaskRunner
, 
ProcessPoolTaskRunner
, or another runner - whether these are regular tasks or mapped tasks - one example flow where tasks execute successfully but don’t appear in the DAG
a
@Marvin We are using 3.7.4 everywhere. You can be concise with your replies. > Inspect a flow run’s task runs in the DB / API How do I get the task run IDs when I only see the flow run? > Check worker / flow logs for task-run event or state-recording issues fatal: not a git repository (or any of the parent directories): .git Apart from this, everything looks clean. I see 
creating task run
... logs.
m
✅ thought for 159.3 seconds
Yes — easiest way is via the Python client or REST API. Python
Copy code
python
import asyncio
from prefect.client.orchestration import get_client
from prefect.client.schemas.filters import TaskRunFilter, TaskRunFilterFlowRunId

FLOW_RUN_ID = "your-flow-run-id"

async def main():
    async with get_client() as client:
        task_runs = await client.read_task_runs(
            task_run_filter=TaskRunFilter(
                flow_run_id=TaskRunFilterFlowRunId(any_=[FLOW_RUN_ID])
            )
        )
        for tr in task_runs:
            print(tr.id, tr.name, tr.state_type, tr.start_time)

asyncio.run(main())
REST API
Copy code
bash
curl -X POST "$PREFECT_API_URL/task_runs/filter" \
  -H "Authorization: Bearer $PREFECT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_runs": {
      "flow_run_id": { "any_": ["<FLOW_RUN_ID>"] }
    }
  }'
A very important detail: If the “missing tasks” are actually tasks that launch subflows, then in 3.7 those parent task runs are intentionally hidden from the DAG, and you only see the subflow run node. That matches your symptom exactly: subflow runs show, task nodes do not. So first question: - Are the missing nodes ordinary `@task`s? - Or are they `@task`s / calls that wrap or invoke 
@flow
? Also, 
fatal: not a git repository
is probably unrelated unless your code explicitly shells out to git. If you want, paste one minimal example of a “missing task” and I can tell you whether it’s expected DAG behavior or a bug.
a
@Marvin They are all normal 
@task
s. When I tried doing curl for task runs. On the main flow, the api returns, only the subflow runs, but no task runs. On the subflows, the api returns nothing because they only have tasks in them.
m
✅ thought for 172.1 seconds
That strongly suggests a task-run persistence problem, not just a UI issue. If 
/task_runs/filter
returns no normal task runs, then Prefect is not recording them in the API/database. Most likely causes in 3.7.4: 1. Task run events are not being persisted Prefect 3 records task runs through an event/recorder pipeline. If that recorder is unhealthy or disabled, you can see execution logs like 
creating task run
, but nothing shows up via API. 2. Only subflow-invocation task runs are making it through That fits what you're seeing: parent flow shows the subflow launcher task runs, but ordinary tasks inside subflows never get persisted. Please check these: Is the task run recorder service enabled/running? - If self-hosted server, verify the 
task_run_recorder
service is healthy. - If you have server logs, search for 
task_run_recorder
. Check these settings on the server side
Copy code
bash
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
If you set any custom server service settings, that’s the first thing I’d inspect. Wait a few seconds, then query again There is batching/flush delay, so retry after ~5–10 seconds. Query one known subflow run ID directly Use the subflow’s flow run id and filter task runs by that exact ID again. If still empty after delay, those task runs are not being persisted at all. At this point, the most useful next step is to determine whether you're using: - Prefect Cloud or self-hosted server - any custom task runner / distributed task execution mode Because if this is self-hosted, I’d focus immediately on the server-side recorder service. If it’s Cloud, this starts to look more like a 3.7.4 regression or an execution-mode-specific bug. If you want, I can give you the one fastest command/check for either: - Prefect Cloud, or - self-hosted server.
a
> 130802.906 | WARNING | prefect_aws.observers.ecs - SQS queue 'prefect-ecs-tasks-events' does not exist in region 'default'. This worker will continue to submit ECS tasks, but event replication and crash detection will not work. To enable ECS event replication and crash detection, deploy an SQS queue using 
prefect-aws ecs-worker deploy-events
and configure the PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME environment variable on your worker to point to the deployed queue. @Marvin Does it have to do anything with this warning?
m
✅ thought for 14.7 seconds
Probably not for missing task runs. That warning is about ECS infrastructure event replication / crash detection, not normal Prefect task-run persistence. So: - Yes, you should fix it if you want better ECS crash detection - No, it is unlikely to be the reason ordinary 
@task
runs are absent from 
/task_runs/filter
What that warning affects: - ECS task state replication - detecting crashed/killed ECS jobs - infrastructure-level events What your symptom points to instead: - Prefect task runs are being created in execution logs - but not persisted/read back from the API That is a different path. You can fix the ECS warning by following the worker message: - deploy the queue with the 
prefect-aws
ECS worker helper - set 
PREFECT_INTEGRATIONS_AWS_ECS_OBSERVER_SQS_QUEUE_NAME
Docs: - prefect-aws integration docs Given your symptoms, I’d focus next on: 1. whether you are on Prefect Cloud or self-hosted server 2. whether the task run recorder service is healthy 3. whether this reproduces with a tiny flow of just: - one parent flow - one subflow - two plain `@task`s inside the subflow If you tell me Cloud vs self-hosted, I’ll give you the shortest next check.
a
@Marvin it is self-hosted
m
✅ thought for 12.6 seconds
Then yes: for self-hosted, the next thing to check is the task run recorder on the server. Most likely check Search your server logs for: - 
task_run_recorder
- 
record_bulk_task_run_events
- 
IntegrityError
- 
dropped
- 
retry
Also verify this is enabled on the server:
Copy code
bash
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED=true
If you run Prefect server yourself, also wait a few seconds after the flow finishes before querying 
/task_runs/filter
, since recording is batched. What I’d do next, in order: 1. Confirm recorder enabled
Copy code
bash
echo $PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED
2. Inspect server logs Look for any task-run-recorder errors/warnings around the time the flow ran. 3. Restart the server services If the recorder got wedged, a restart may temporarily restore task-run persistence. 4. Run a tiny repro flow A flow with: - one 
@task
- one subflow - one 
@task
inside the subflow Then compare: - 
/flow_runs/filter
- 
/task_runs/filter
If you want, I can give you a very short repro flow plus the exact 
curl
calls to validate recorder behavior.
a
@Marvin I do not see those strings in the server logs (prefect server start) or in worker (prefect worker start). Also, PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED variable is not set.
m
✅ thought for 14.3 seconds
That’s normal — if the env var is unset, it should default to enabled. So the next useful conclusion is: - this is not obviously “recorder disabled” - and the ECS SQS warning is likely unrelated Since 
/task_runs/filter
shows only subflow launcher task runs and no ordinary tasks, I’d treat this as either: - a 3.7.4 self-hosted bug/regression, or - something specific about how those tasks are being executed Fastest isolation step: run this minimal repro on the same setup.
Copy code
python
from prefect import flow, task

@task
def t1():
    return 1

@task
def t2(x):
    return x + 1

@flow
def child():
    x = t1()
    t2(x)

@flow
def parent():
    t1()
    child()
    t2(5)

if __name__ == "__main__":
    parent()
Then check: - does the parent flow show only the child subflow node? - do any plain tasks appear in 
/task_runs/filter
? If no plain tasks appear even in this minimal case, that strongly points to a server/version issue rather than your code. At that point, I’d recommend either: 1. upgrade to the latest available 3.7.x / current 3.x, or 2. open/check a GitHub issue because this does not look like expected behavior Releases: - Prefect releases - Prefect issues If you want, I can do one quick pass through existing GitHub issues/discussions for this exact symptom.
a
@Marvin This is one of the minimal flow runs we have, with a single task Log is here from the flow run 45518.163 | INFO | prefect.runner.workspace_resolver - Downloading flow code from storage at '/app' 145520.453 | INFO | Flow run 'chi6-matalas-x' - Beginning flow run 'chi6-matalas-x' for flow 'mesh-test-flow' 145520.468 | INFO | Flow run 'chi6-matalas-x' - View at abcde.com/runs/flow-run/07d6791b-86e4-4a43-a15b-8b2dc632605e 145520.512 | INFO | Task run 'task_update_test_environment-67e' - Updating workflow testing environment json file 145526.568 | INFO | Task run 'task_update_test_environment-67e' - JSON file created: /data/test_outputs/files.json 145526.573 | INFO | Task run 'task_update_test_environment-67e' - Finished in state Completed() 145527.573 | INFO | Flow run 'chi6-matalas-x' - Finished in state Completed()
m
✅ thought for 19.2 seconds
Thanks — that log is very helpful. This confirms the task really is executing normally: - task starts - task finishes 
Completed()
- flow finishes 
Completed()
So this is not a “task never ran” problem. If that task still does not appear in 
/task_runs/filter
, then the likely problem is: the task run exists in memory during execution, but is not being persisted to the API/database in your self-hosted 3.7.4 setup. Given that this is a single normal task, I would stop looking at ECS/SQS and focus on the server/API layer. Two fast checks: 1. Query task runs by flow-run id again after completion with a delay Wait 10–15 seconds after the flow completes, then call:
Copy code
bash
curl -X POST "$PREFECT_API_URL/task_runs/filter" \
  -H "Authorization: Bearer $PREFECT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task_runs": {
      "flow_run_id": { "any_": ["07d6791b-86e4-4a43-a15b-8b2dc632605e"] }
    }
  }'
If still empty, persistence is failing. 2. Check the database directly If you can access the DB, run:
Copy code
sql
select id, flow_run_id, name, state_type, start_time
from task_run
where flow_run_id = '07d6791b-86e4-4a43-a15b-8b2dc632605e';
Interpretation: - rows exist in DB -> API/filter/UI issue - no rows in DB -> server never persisted task runs At this point, based on everything you shared, my best guess is: - this is a self-hosted 3.7.4 regression/bug affecting task-run persistence Since GitHub search is failing on my side right now, I can’t verify an existing issue, but your reproduction is strong. My practical recommendation: - upgrade your self-hosted server/API to the latest available 3.x first - if you cannot, pin back to the version that worked - and if needed, open a GitHub issue with exactly this repro: - self-hosted - 3.7.4 - ECS workers - normal 
@task
executes and logs 
Completed()
- 
/task_runs/filter
returns no task runs for the flow run - subflow launcher task runs may appear, ordinary task runs do not Issue tracker: - Prefect issues If you want, I can help you draft the issue text in 5–6 lines.
a
@Marvin No rows in db. Can you help make an issue
m
✅ thought for 14.7 seconds
Yes — here’s a concise issue draft you can paste into GitHub issues. Title 
Task runs execute normally but are not persisted in self-hosted Prefect 3.7.4
Body
Copy code
text
### Bug summary

In a self-hosted Prefect 3.7.4 deployment, normal `@task` runs execute and log successfully, but no corresponding rows are written to the `task_run` table and `/task_runs/filter` returns no task runs.

Subflow runs do appear, but ordinary task runs do not.

### Version info

- Prefect version: 3.7.4
- Deployment type: self-hosted
- Workers: ECS workers on AWS
- Server started with: `prefect server start`

### What I expected

Normal `@task` runs should be persisted and visible:
- in the `task_run` table
- via `/task_runs/filter`
- in the run DAG

### What happened

Task logs show normal execution and completion, for example:

```text
14:55:20.453 | INFO    | Flow run 'chi6-matalas-x' - Beginning flow run 'chi6-matalas-x' for flow 'mesh-test-flow'
14:55:20.512 | INFO    | Task run 'task_update_test_environment-67e' - Updating workflow testing environment json file
14:55:26.568 | INFO    | Task run 'task_update_test_environment-67e' - JSON file created: /data/test_outputs/files.json
14:55:26.573 | INFO    | Task run 'task_update_test_environment-67e' - Finished in state Completed()
14:55:27.573 | INFO    | Flow run 'chi6-matalas-x' - Finished in state Completed()
However: - 
POST /task_runs/filter
for that flow run returns no normal task runs - querying the DB shows no rows in 
task_run
for that 
flow_run_id
### Minimal example
Copy code
python
from prefect import flow, task

@task
def t1():
    print("hello")

@flow
def test_flow():
    t1()

if __name__ == "__main__":
    test_flow()
### Additional notes - This worked in 3.4.10 - After upgrading to 3.7.x, only subflow runs show in the DAG - Ordinary task runs are missing from both the UI and API - 
PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED
is not set explicitly - I do see this ECS warning, but it seems unrelated:
Copy code
text
SQS queue 'prefect-ecs-tasks-events' does not exist in region 'default'. This worker will continue to submit ECS tasks, but event replication and crash detection will not work.
### Verification performed - Waited after flow completion before querying - Queried 
/task_runs/filter
by 
flow_run_id
- Checked the 
task_run
table directly: no rows exist for affected flow runs - Server logs do not obviously show 
task_run_recorder
errors ``` If you want, I can also make this shorter and more maintainer-friendly with a “Steps to reproduce / Actual / Expected” format.
a
Error was, I did not have prefect server services running. Now TaskRunRecorder is running and task runs are coming into the api as expected. In the server howerver, I see an error, but everything is running as expected despite the error log.
174408.767 | ERROR | prefect.server.utilities.postgres_listener - Failed to establish raw asyncpg connection for LISTEN/NOTIFY: bad query field: 'EO|Bs8$ctC@prefect.abcdef.us-east-1.rds.amazonaws.com:5432/postgres'
Seems like it has to do with our PG Password:
'postgresql+asyncpg://prefect:D_~b$yy?EO|Zs8$ctC@prefect.abcdef....
See how everything before 
?
has been ignored. prefect server serivces ls, Available Services │ TaskRunRecorder │ ✓ PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED │ EventLogger │ x PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED │ EventPersister │ ✓ PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED │ ReactiveTriggers │ ✓ PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED │ Actions │ ✓ PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED │ Distributor │ ✓ PREFECT_API_EVENTS_STREAM_OUT_ENABLED │ LogDistributor │ x PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED Any insight? @Marvin
m
✅ thought for 17.0 seconds
Yes — that error is very likely due to an unescaped special character in the Postgres password inside the connection URL. In a URL, 
?
starts the query string, so a password like:
Copy code
text
D_~b$yy?EO|Zs8$ctC
must be percent-encoded before being embedded in:
Copy code
text
<postgresql+asyncpg://user:password@host:5432/dbname>
Otherwise Prefect / SQLAlchemy / asyncpg may parse everything after 
?
as URL query content instead of password. That explains this error:
Copy code
text
Failed to establish raw asyncpg connection for LISTEN/NOTIFY: bad query field ...
because the listener code uses the DB URL and the parsed URL is malformed. Why things still mostly work - normal DB access may still work through one code path - but the raw asyncpg 
LISTEN/NOTIFY
connection used by 
postgres_listener
is stricter and fails Fix Percent-encode the password in the URL. For example, encode special characters like: - 
?
-> 
%3F
- 
|
-> 
%7C
- 
$
-> 
%24
- 
~
usually ok, but encoding is also fine So the password:
Copy code
text
D_~b$yy?EO|Zs8$ctC
should become something like:
Copy code
text
D_~b%24yy%3FEO%7CZs8%24ctC
Then use that in your connection string. Example:
Copy code
text
<postgresql+asyncpg://prefect:D_~b%24yy%3FEO%7CZs8%24ctC@host:5432/postgres>
Best practice If possible: - avoid putting raw credentials directly into URLs - or regenerate a DB password without reserved URL characters Python can encode it with:
Copy code
python
from urllib.parse import quote
print(quote("D_~b$yy?EO|Zs8$ctC", safe=""))
This should help: - remove the 
postgres_listener
error - restore LISTEN/NOTIFY behavior correctly Relevant docs: - Self-hosted Prefect - SQLAlchemy database URLs So short answer: yes, this is almost certainly your password needing URL encoding.
a
@Marvin What is it 
postgres_listener
used for?
m
✅ thought for 7.7 seconds
postgres_listener
is used for Postgres 
LISTEN/NOTIFY
-based signaling inside self-hosted Prefect. In practice, it helps Prefect services react to DB-backed events without polling as much. Most relevant uses are around: - coordinating server-side services - reacting to newly persisted events / notifications - trigger / automation-related wakeups - internal event distribution paths That’s why things can still mostly work even when it errors: - core API/database operations can still succeed - but some reactive/background behavior may be degraded or less efficient So if 
postgres_listener
is broken, you may see: - noisier logs - delayed or degraded automations/triggers/event handling - fallback behavior instead of efficient notifications But it usually does not block basic flow/task execution by itself. Given your service list, this is most relevant to: - 
ReactiveTriggers
- 
Actions
- event-related background services So I would still fix it, even if runs look healthy.
a
@Marvin With password encoded, connection fails with 
InvalidAuthorizationSpecificationError
. My connection URL has changed to > Set 'PREFECT_API_DATABASE_CONNECTION_URL' to > 'postgresql+asyncpg://prefect:H_~b%24yy%3FEO%7CBs8%24ctC@prefect.abcde.us-east-1.rds.amazonaws.com:5432/postgres'.
m
✅ thought for 17.2 seconds
The encoded password is right, but your URL format is wrong. You have:
Copy code
text
<postgresql+asyncpg://prefect:H_~b%24yy%3FEO%7CBs8%24ctC>...
It should be:
Copy code
text
<postgresql+asyncpg://prefect:H_~b%24yy%3FEO%7CBs8%24ctC@host:5432/postgres>
Wait — actually the key thing is whether there is an 
@
after the password. Valid URL shape is:
Copy code
text
postgresql+asyncpg://<username>:<password>@<host>:<port>/<database>
So check all of these carefully: 1. Username/password separator Must be 
username:password
2. Password/host separator Must be 
@
3. No extra quotes/brackets The env var value should just be the raw URL Correct example:
Copy code
text
<postgresql+asyncpg://prefect:H_~b%24yy%3FEO%7CBs8%24ctC@prefect.abcdef.us-east-1.rds.amazonaws.com:5432/postgres>
If you already have that format and still get 
InvalidAuthorizationSpecificationError
, then the next likely causes are: - username is wrong - database name is wrong - password was encoded twice - password in RDS is actually different from what you think One subtlety: - only encode the password portion - do not encode the whole URL You can verify with Python:
Copy code
python
from urllib.parse import quote

password = "H_~b$yy?EO|Bs8$ctC"
print(quote(password, safe=""))
Then place that output only into:
Copy code
text
<postgresql+asyncpg://prefect:<encoded_password>@host:5432/postgres>
If you want, paste the URL with host partially redacted like:
Copy code
text
<postgresql+asyncpg://prefect:***@host:5432/postgres>
and I can sanity-check the structure.
a
@Marvin With password encoded, connection fails with 
InvalidAuthorizationSpecificationError
.
asyncpg.exceptions.InvalidAuthorizationSpecificationError: no pg_hba.conf entry for host "172.172.17.17", user "prefect", database "postgres", no encryption
ERROR: Application startup failed. Exiting.
My connection URL has changed to
Set 'PREFECT_API_DATABASE_CONNECTION_URL' to
'postgresql+asyncpg://prefect:H_~b%24yy%3FEO%7CBs8%24ctC@prefect.abcde.us-east-1.rds.amazonaws.com:5432/postgres'.
m
✅ thought for 10.6 seconds
This is now a different issue. Your URL is still malformed in the Slack paste, but the actual error:
Copy code
text
no pg_hba.conf entry for host "...", user "prefect", database "postgres", no encryption
means: the server is requiring SSL/TLS, but the asyncpg connection used by Prefect’s postgres listener is connecting without encryption. So: - password encoding was needed - but now the blocker is SSL mode / TLS requirements For RDS, you usually need SSL enabled. Also, your URL should be structurally:
Copy code
text
<postgresql+asyncpg://prefect:<encoded_password>@prefect.abcde.us-east-1.rds.amazonaws.com:5432/postgres?ssl=require>
A few important notes: 1. No angle brackets / Slack mailto formatting The real env var must be plain text only. 2. The username should not be outside the authority It should be:
Copy code
text
   <postgresql+asyncpg://user:pass@host:port/db>
not
Copy code
text
   <postgresql+asyncpg://user>>:pass...
3. Add SSL requirement For RDS, try:
Copy code
text
   ?ssl=require
So the likely correct value is:
Copy code
text
<postgresql+asyncpg://prefect:H_~b%24yy%3FEO%7CBs8%24ctC@prefect.abcde.us-east-1.rds.amazonaws.com:5432/postgres?ssl=require>
If that still fails, use the RDS CA / SSL config appropriate for your environment, but 
ssl=require
is the first thing to try. So short version: - your earlier problem was password parsing - your current problem is TLS required by Postgres/RDS - add 
?ssl=require
to a correctly formed URL Relevant docs: - SQLAlchemy database URLs - PostgreSQL SSL support - RDS PostgreSQL SSL
4 Views
Open in Slack
PreviousNext
Powered by