Hi all, not sure this belongs here. How are you dealing with non-descriptive error messages when starting a flow-run?
All I get is "Submission failed" and no error message (see img). I've got the work queue agent set to debug and it's not showing anything out of the ordinary. It fails also pretty inconsistently, like every other time.
01/03/2023, 10:07 AM
I don't think the agent logs will show up for flows / tasks runs.
Depending on the logs are configured / store, you should be able to access them via your cloud setup.
For example, if your agent is an AWS ECS Service configured with cloud watch, you can find those logs in the cloud watch console. I'm gonna assume that it's similar for GCP, Azure etc.
01/03/2023, 10:25 AM
Appreciate the response @Kelvin DeCosta
The agent is local so not sure where I would debug this or why it's failing to submit the run to the agent. (to be clear the agent is a on-premise Ubuntu server). I'm using Prefect Cloud v2..
To me it looks like PrefectUI is failing to trigger any communication with the agent sporadically.
01/03/2023, 2:41 PM
Ah okay! Do you have access to the standard output / log stream generated by the agent process?
01/03/2023, 2:50 PM
Yes. it just shows the polling of scheduled/cancelled tasks constantly and I have to trigger the run of the deployment a couple of times and sometimes it works sometimes is just gives "Submission failed" and nothing is shown on either side.
@Kelvin DeCosta Just wanted to quickly update why this was occurring.
We seemed to be dealing with a ghost agent, we killed all instances of agents but still the work-queue was "Healthy". A swift reboot seemed to fix the problem.
We are also running Prefect v1 agents on the same machine, not sure if it relates to that.