Hi i have an bash script which starts prefect worker queues

Question

Hi i have an bash script which starts prefect worker queues prefect agent start q name every hour and for some reason one of the agent says its unhealthy most of the time any reasons what could the issues my understanding was there was no responce from that queue but i am more curious as to whats causing this issue and that specific queue alone and one of my flow runs crashed and it threw and 500 responce error it says it was internal server error any leads on this would help I am using prefect 2 with server running on amazon ec2 instance and i use prefect cloud if that helps thanks in advance

Revanth · Answer

was going through the logs and found this ERROR | prefect agent Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED h2 exceptions ProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED

Mason Menges · Answer

Hey < Revanth> nothing immediately comes to mind here but would you mind providing some more details about the flow that triggered this error what task runner are you using How is the flow being triggered i e is this running on a schedule or is it being triggered as part of a different flow run do you have an example of the flow code

Revanth · Answer

hey mason thanks for reaching out i am using ConcurrentTaskRunner and the flow is triggered through schedule and i am using Deployment build from flow and passing in the function

Revanth · Answer

the error raises when i run this bash script prefect agent start q work queue 1 amp prefect agent start q work queue 2 amp prefect agent start q work queue 3 amp amp fg it throws the error which i mentioned in the messages

Revanth · Answer

`Agent started Looking for work from queue s user analytics computation queue ` 07 37 28 146 | ERROR | prefect agent Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED Traceback most recent call last File home ubuntu local lib python3 10 site packages h2 connection py line 224 in process input func target state = self transitions self state input KeyError lt ConnectionState CLOSED 3 gt lt ConnectionInputs RECV PING 14 gt During handling of the above exception another exception occurred Traceback most recent call last File home ubuntu local lib python3 10 site packages httpcore async http2 py line 114 in handle async request status headers = await self receive response File home ubuntu local lib python3 10 site packages httpcore async http2 py line 231 in receive response event = await self receive stream event request stream id File home ubuntu local lib python3 10 site packages httpcore async http2 py line 262 in receive stream event await self receive events request stream id File home ubuntu local lib python3 10 site packages httpcore async http2 py line 283 in receive events events = await self read incoming data request File home ubuntu local lib python3 10 site packages httpcore async http2 py line 345 in read incoming data events = self h2 state receive data data File home ubuntu local lib python3 10 site packages h2 connection py line 1463 in receive data events extend self receive frame frame File home ubuntu local lib python3 10 site packages h2 connection py line 1487 in receive frame frames events = self frame dispatch table frame class frame File home ubuntu local lib python3 10 site packages h2 connection py line 1760 in receive ping frame events = self state machine process input File home ubuntu local lib python3 10 site packages h2 connection py line 228 in process input raise ProtocolError h2 exceptions ProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED During handling of the above exception another exception occurred Traceback most recent call last File home ubuntu local lib python3 10 site packages httpx transports default py line 60 in map httpcore exceptions yield File home ubuntu local lib python3 10 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File home ubuntu local lib python3 10 site packages httpcore async connection pool py line 253 in handle async request raise exc File home ubuntu local lib python3 10 site packages httpcore async connection pool py line 237 in handle async request response = await connection handle async request request File home ubuntu local lib python3 10 site packages httpcore async connection py line 90 in handle async request return await self connection handle async request request File home ubuntu local lib python3 10 site packages httpcore async http2 py line 144 in handle async request raise LocalProtocolError exc ` pragma nocover` `httpcore LocalProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED` `The above exception was the direct cause of the following exception ` `Traceback most recent call last ` `File home ubuntu local lib python3 10 site packages prefect agent py line 154 in get and submit flow runs` `queue runs = await self client get runs in work queue ` `File home ubuntu local lib python3 10 site packages prefect client orion py line 759 in get runs in work queue` `response = await ` `File home ubuntu local lib python3 10 site packages httpx client py line 1842 in post` `return await self request ` `File home ubuntu local lib python3 10 site packages httpx client py line 1527 in request` `return await self send request auth=auth follow redirects=follow redirects ` `File home ubuntu local lib python3 10 site packages prefect client base py line 159 in send` `await super send args kwargs ` `File home ubuntu local lib python3 10 site packages httpx client py line 1614 in send` `response = await self send handling auth ` `File home ubuntu local lib python3 10 site packages httpx client py line 1642 in send handling auth` `response = await self send handling redirects ` `File home ubuntu local lib python3 10 site packages httpx client py line 1679 in send handling redirects` `response = await self send single request request ` `File home ubuntu local lib python3 10 site packages httpx client py line 1716 in send single request` `response = await transport handle async request request ` `File home ubuntu local lib python3 10 site packages httpx transports default py line 352 in handle async request` `with map httpcore exceptions ` `File usr lib python3 10 contextlib py line 153 in exit ` `self gen throw typ value traceback ` `File home ubuntu local lib python3 10 site packages httpx transports default py line 77 in map httpcore exceptions` `raise mapped exc message from exc` `httpx LocalProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED` `07 37 32 732 | ERROR | prefect agent Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED` `Traceback most recent call last ` `File home ubuntu local lib python3 10 site packages h2 connection py line 224 in process input` `func target state = self transitions self state input ` `KeyError lt ConnectionState CLOSED 3 gt lt ConnectionInputs RECV PING 14 gt ` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File home ubuntu local lib python3 10 site packages httpcore async http2 py line 114 in handle async request` `status headers = await self receive response ` `File home ubuntu local lib python3 10 site packages httpcore async http2 py line 231 in receive response` `event = await self receive stream event request stream id ` `File home ubuntu local lib python3 10 site packages httpcore async http2 py line 262 in receive stream event` `await self receive events request stream id ` `File home ubuntu local lib python3 10 site packages httpcore async http2 py line 283 in receive events` `events = await self read incoming data request ` `File home ubuntu local lib python3 10 site packages httpcore async http2 py line 345 in read incoming data` `events = self h2 state receive data data ` `File home ubuntu local lib python3 10 site packages h2 connection py line 1463 in receive data` `events extend self receive frame frame ` `File home ubuntu local lib python3 10 site packages h2 connection py line 1487 in receive frame` `frames events = self frame dispatch table frame class frame ` `File home ubuntu local lib python3 10 site packages h2 connection py line 1760 in receive ping frame` `events = self state machine process input ` `File home ubuntu local lib python3 10 site packages h2 connection py line 228 in process input` `raise ProtocolError ` `h2 exceptions ProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File home ubuntu local lib python3 10 site packages httpx transports default py line 60 in map httpcore exceptions` `yield` `File home ubuntu local lib python3 10 site packages httpx transports default py line 353 in handle async request` `resp = await self pool handle async request req ` `File home ubuntu local lib python3 10 site packages httpcore async connection pool py line 253 in handle async request` `raise exc` `File home ubuntu local lib python3 10 site packages httpcore async connection pool py line 237 in handle async request` `response = await connection handle async request request ` `File home ubuntu local lib python3 10 site packages httpcore async connection py line 90 in handle async request` `return await self connection handle async request request ` `File home ubuntu local lib python3 10 site packages httpcore async http2 py line 144 in handle async request` `raise LocalProtocolError exc pragma nocover ` `httpcore LocalProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED` `The above exception was the direct cause of the following exception ` `Traceback most recent call last ` `File home ubuntu local lib python3 10 site packages prefect agent py line 154 in get and submit flow runs` `queue runs = await self client get runs in work queue ` `File home ubuntu local lib python3 10 site packages prefect client orion py line 759 in get runs in work queue` `response = await ` `File home ubuntu local lib python3 10 site packages httpx client py line 1842 in post` `return await self request ` `File home ubuntu local lib python3 10 site packages httpx client py line 1527 in request` `return await self send request auth=auth follow redirects=follow redirects ` `File home ubuntu local lib python3 10 site packages prefect client base py line 159 in send` `await super send args kwargs ` `File home ubuntu local lib python3 10 site packages httpx client py line 1614 in send` `response = await self send handling auth ` `File home ubuntu local lib python3 10 site packages httpx client py line 1642 in send handling auth` `response = await self send handling redirects ` `File home ubuntu local lib python3 10 site packages httpx client py line 1679 in send handling redirects` `response = await self send single request request ` `File home ubuntu local lib python3 10 site packages httpx client py line 1716 in send single request` `response = await transport handle async request request ` `File home ubuntu local lib python3 10 site packages httpx transports default py line 352 in handle async request` `with map httpcore exceptions ` `File usr lib python3 10 contextlib py line 153 in exit ` `self gen throw typ value traceback ` `File home ubuntu local lib python3 10 site packages httpx transports default py line 77 in map httpcore exceptions` `raise mapped exc message from exc` `httpx LocalProtocolError Invalid input ConnectionInputs RECV PING in state ConnectionState CLOSED` `07 37 33 169 | ERROR | prefect agent Invalid input ConnectionInputs SEND HEADERS in state ConnectionState CLOSED` `Traceback most recent call last ` `File home ubuntu local lib python3 10 site packages h2 connection py line 224 in process input` `func target state = self transitions self state input ` `KeyError lt ConnectionState CLOSED 3 gt lt ConnectionInputs SEND HEADERS 0 gt ` `During handling of the above exception another exception occurred `

Revanth · Answer

looks like <https github com PrefectHQ prefect issues 7442> this issue

Mason Menges · Answer

Brilliant Yeah we seen this come up a few times recently and this definitely seems like it could be related smile

Revanth · Answer

not really sure about issue tried reading through bunch of open issue on github but all of them went over my head for an work around wrote few bash scripts which would start the work queue when the flow was supposed to run and pause the work queue after a while i am hopping this would help me solve the issue any other methods would you recommend

Mason Menges · Answer

For clarification you re not starting an agent as part of the flow run correct

Revanth · Answer

not part of the flow run

Revanth · Answer

i am using CronSchedule to schedule my flows the bash scripts are to start the agent when the flow was supposed to run example let us i have scheduled my flow run at 12 am what i am currently doing is running an bash script prefect agent start q work queue name at 11 55 and assumming the flow ends in 10 minutes so after 10 minutes i am running an another bash script prefect work queue pause work queue name hope i gave some useful info sorry english isnt my first language

Mason Menges · Answer

Hey < Revanth> I think this might be related to this issue