Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello,
I am noticing on Prefect `3.0.1` when I instantiate a flow more frequently than not the first task does not display on the graph even though the logs indicate the task was created. Further more, I see 404 errors in the console at the path `/api/task_runs/{task_id}` . When I check the prefect database (postgres) I can confirm that the `task_run` table also does not have a record of the `task_id` .

The workers are ECS tasks
Prefect Version 3.0.1

Hey, are you using prefect cloud or a self hosted server?

We've been <https://github.com/PrefectHQ/prefect/issues/15153|investigating> this for some self hosted users and I've had some trouble reproducing. If you're able to reproduce it reliably would you be able to do the following:

1. Confirm that your client and server are both on `3.0.1`
2. Set `PREFECT_DEBUG_MODE=1` on your client and server
3. Set `PREFECT_LOGGING_SERVER_LEVEL=DEBUG` on your server (and restart so it takes effect)
4. Running a very simple task and tracking the expected <https://github.com/PrefectHQ/prefect/issues/15153#issuecomment-2330149400|logs client side> and <https://github.com/PrefectHQ/prefect/issues/15153#issuecomment-2338446514|server side >

When you say client do you mean the worker itself ?

Ah sorry I mean the execution environment where you task runs. So inside of your ECS Tasks, that your worker is deploying

happy to communicate in this thread or on the issue if it's easier to post your findings.

If everything is working as expected you should be able to track a task run event all the way through, if something is wrong I'd expect either some specific logs missing or some errors (hopefully)

Thanks in advance for looking into this with me :pray:

Sure thing thank you, I’ll set the debug modes and start digging

appreciate it. happy to answer any questions as well if I can help

I am still looking into the client and server side logs but I can confirm that websockets are configured as they should be.

I did notice that this is a non issue locally on my mac. This is only an issue running prefect server and the workers and flows with ECS tasks.

Thanks for following up! Apologies for the delayed response, I was out yesterday.

 That is interesting. So if you run a flow against your server locally everything processes? But if you then run a flow on ECS, the runs don't show up? And when the flow runs execute on ECS they're definition version 3.0.1?

and just to confirm _task run events_ are still being sent from the flow run execution environments successfully? So nothing like `ERROR - EventsWorker failed with n pending items` and you can view those in the UI? but you can't view task runs themselves

The strange thing is I see the events showing up in the event feed.  

But when I take the task I’d from the event feed and go to the task run url in the browser the task does not show up.  No task pulls up 

So strange :thinking_face:. Okay two follow ups for you:

• Were you able to enable `PREFECT_LOGGING_SERVER_LEVEL=DEBUG` on the server and track the task run event server side being received and sent to the `TaskRunRecorder` service? or any errors?
• Would you be able to share in this chat a copy of the full task run event jsons:
    ◦ one generated locally
    ◦ one generated on ECS