hi everyone im facing a weird issue in flow runs im running

Question

hi everyone im facing a weird issue in flow runs im running prefect on aws eks self hosted instance of orion and kubernetes jobs on different namespaces 1 i have 9 running flows which i can see on the ui but i cant find the corresponding job on my cluster 2 when i check the concurrency limit for the tags that these flow use i can see 14 task runs these flow have parallel task runs so some of them are having more than one task stuck in running state so basically 9 running flows with 14 running tasks as i can see on the ui but i cant find the jobs on my cluster doubt how did these flow runs get terminated on the cluster and even if they did why did it not update in the ui or lets say why was the state not updated in meta db attaching screenshots for the same

Deepanshu Aggarwal · Answer

9 running flows

Deepanshu Aggarwal · Answer

14 concurrent running tasks

Deepanshu Aggarwal · Answer

no jobs found

redsquare · Answer

What do the agent logs say

Deepanshu Aggarwal · Answer

```07 00 48 366 | ERROR | prefect agent lt ConnectionTerminated error code ErrorCodes NO ERROR last stream id 19999 additional data None gt ```

Deepanshu Aggarwal · Answer

```Traceback most recent call last File usr local lib python3 9 site packages h2 connection py line 224 in process input func target state = self transitions self state input KeyError lt ConnectionState CLOSED 3 gt lt ConnectionInputs SEND HEADERS 0 gt During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 9 site packages httpcore async http2 py line 108 in handle async request await self send request headers request=request stream id=stream id File usr local lib python3 9 site packages httpcore async http2 py line 205 in send request headers self h2 state send headers stream id headers end stream=end stream File usr local lib python3 9 site packages h2 connection py line 766 in send headers self state machine process input ConnectionInputs SEND HEADERS File usr local lib python3 9 site packages h2 connection py line 228 in process input raise ProtocolError h2 exceptions ProtocolError Invalid input ConnectionInputs SEND HEADERS in state ConnectionState CLOSED```

Deepanshu Aggarwal · Answer

```During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 9 site packages httpx transports default py line 60 in map httpcore exceptions yield File usr local lib python3 9 site packages httpx transports default py line 353 in handle async request resp = await self pool handle async request req File usr local lib python3 9 site packages httpcore async connection pool py line 253 in handle async request raise exc File usr local lib python3 9 site packages httpcore async connection pool py line 237 in handle async request response = await connection handle async request request File usr local lib python3 9 site packages httpcore async connection py line 90 in handle async request return await self connection handle async request request File usr local lib python3 9 site packages httpcore async http2 py line 141 in handle async request raise RemoteProtocolError self connection error event httpcore RemoteProtocolError lt ConnectionTerminated error code ErrorCodes NO ERROR last stream id 19999 additional data None gt ```

redsquare · Answer

Possibly erroring trying to speak to k8s api to create the jobs

Deepanshu Aggarwal · Answer

but jobs did get created right cz like 98 of the tasks ran it must have terminated in the middle somewhere without updating the state

redsquare · Answer

The jobs should still be there

redsquare · Answer

if they got created

Christopher Boyd · Answer

Are they being created in that namespace or another

Christopher Boyd · Answer

what if you run kubectl get pods A

Deepanshu Aggarwal · Answer

In the selected namespace

Deepanshu Aggarwal · Answer

I mean the job has definitely terminated because it s not something that would take more than 30min at max but has been in running state for somewhat 24 hours now

Deepanshu Aggarwal · Answer

i think the issue is somehow the errors that task run into which should mark the task failed are not getting updated in the metadb

Deepanshu Aggarwal · Answer

and neither are they coming up in logs on the ui

Deepanshu Aggarwal · Answer

see the last line in the logs but this is not reflected in the ui

Taylor Curran · Answer

< Deepanshu Aggarwal> code here

Taylor Curran · Answer

```from prefect import flow task from datetime import date timedelta from prefect dask import DaskTaskRunner from prefect deployments import run deployment task def ingest raw customers dataset size pass task def ingest raw payments dataset size pass task def ingest raw orders dataset size start date end date pass Parallel flow task runner=DaskTaskRunner def partition ingestion async dataset size start date end date n partitions date delta = end date start date partition period = date delta n partitions for i in range n partitions end date = start date + partition period start date = start date+partition period ingest raw orders submit start date end date dataset size return Done Async flow def partition ingestion async dataset size start date end date n partitions date delta = end date start date partition period = date delta n partitions start dates = end dates = for i in range n partitions start dates append start date end dates append start date + partition period start date = start date+partition period ingest raw orders map start dates end dates dataset size return Done k8s or separate container flow def sub flow print subflow flow def partition ingestion unique pods dataset size start date end date n partitions date delta = end date start date partition period = date delta n partitions for i in range n partitions end date = start date + partition period start date = start date+partition period run deployment deployment= ingest raw orders k8s deployment parameters= start date start date end date end date dataset size dataset size flow retries=3 retry delay seconds=30 def raw data jaffle shop start date date = date 2022 2 1 end date date = date today dataset size int = 10 000 parametrized for backfills n partitions int = 3 ingest raw customers submit dataset size ingest raw payments submit dataset size partition ingestion async dataset size start date end date n partitions if name == main raw data jaffle shop date 2022 2 1 date today 1000 4 ```