Hello, I am noticing on Prefect

3.0.1

when I instantiate a flow more frequently than not the first task does not display on the graph even though the logs indicate the task was created. Further more, I see 404 errors in the console at the path

/api/task_runs/{task_id}

. When I check the prefect database (postgres) I can confirm that the

task_run

table also does not have a record of the

task_id

. The workers are ECS tasks Prefect Version 3.0.1

Hey, are you using prefect cloud or a self hosted server?

Self hosted

We've been investigating this for some self hosted users and I've had some trouble reproducing. If you're able to reproduce it reliably would you be able to do the following: 1. Confirm that your client and server are both on

3.0.1

2. Set

PREFECT_DEBUG_MODE=1

on your client and server 3. Set

PREFECT_LOGGING_SERVER_LEVEL=DEBUG

on your server (and restart so it takes effect) 4. Running a very simple task and tracking the expected logs client side and server side

When you say client do you mean the worker itself ?

Ah sorry I mean the execution environment where you task runs. So inside of your ECS Tasks, that your worker is deploying

Sure thing thank you, I’ll set the debug modes and start digging

appreciate it. happy to answer any questions as well if I can help

I am still looking into the client and server side logs but I can confirm that websockets are configured as they should be. I did notice that this is a non issue locally on my mac. This is only an issue running prefect server and the workers and flows with ECS tasks.

Thanks for following up! Apologies for the delayed response, I was out yesterday. That is interesting. So if you run a flow against your server locally everything processes? But if you then run a flow on ECS, the runs don't show up? And when the flow runs execute on ECS they're definition version 3.0.1?

Yes they are

and just to confirm task run events are still being sent from the flow run execution environments successfully? So nothing like

ERROR - EventsWorker failed with n pending items

and you can view those in the UI? but you can't view task runs themselves

Nope I haven’t seen that

So strange 🤔. Okay two follow ups for you: • Were you able to enable

PREFECT_LOGGING_SERVER_LEVEL=DEBUG

on the server and track the task run event server side being received and sent to the

TaskRunRecorder

service? or any errors? • Would you be able to share in this chat a copy of the full task run event jsons: ◦ one generated locally ◦ one generated on ECS