https://prefect.io logo
s

Samuel Hinton

02/10/2021, 4:08 PM
Having ye old issue of “Flows are submitted and an agent sees them, but they get stuck in ‘submitted for execution’“. I am wondering if anyone knows how to debug this. I know that: • the labels on the task and the agent match • the docker agent has pulled the image, started a container and “completed flow run submission” from the logs • Using either the Local executor or LocalDaskExecutor does not change the fact that: • Flows never progress from the “submitted for execution” state Is there a way of trying to look inside an agent to figure out what is not working?
m

Mariia Kerimova

02/10/2021, 4:18 PM
Hello Samuel! Do you see any logs? Can you pass a flag
--show-flow-logs
to the agent?
s

Samuel Hinton

02/10/2021, 4:18 PM
I can restart the agent with those kwargs, one moment 🙂
Ah yes I see a
Copy code
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='ds-prefect-apollo', port=4200): Max retries exceeded with url: /graphql/graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd8aa3018b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
error when it tries to start. I wonder if why its /graphql/graphql instead of /graphql!
m

Mariia Kerimova

02/10/2021, 4:22 PM
Yes, looks like
j

Josh Greenhalgh

03/17/2021, 12:32 PM
@Samuel Hinton did you ever work out what was going on here? Where
/graphql/graphql
came from? What was causing the connection issues to apollo? I am having very similar issues but no idea why...
My flow runs fine for the majority of its tasks then a few mapped tasks finish in a pending state and I have errors very similar to the one you have above...
s

Samuel Hinton

03/17/2021, 12:34 PM
I actually didnt manage to figure this out at all 😞. I upgraded to the latest version and restarted the services a few times and it started working consistently. The root cause of the issue is still a mystery. But yes, sounds like exactly what I was experiencing but unfortunately no words of wisdom from me