Hey dear community We recently have an issue when we were tr

Question

Hey dear community We recently have an issue when we were triggering 3 instances of the same parent flow that triggers a child flows it s not the common use case but we wanted to test this case also And we get errors from the prefect server that goes like this I will write in the thread What can possibly be the cause Thanks

Noam polak · Answer

error message num 1 Task some child flow Exception encountered during task execution Traceback most recent call last File usr local lib python3 10 site packages prefect engine task runner py line 876 in get task run state value = prefect utilities executors run task with timeout File usr local lib python3 10 site packages prefect utilities executors py line 467 in run task with timeout return task run args kwargs type ignore File usr local lib python3 10 site packages prefect tasks prefect flow run py line 260 in wait for flow run for log in watch flow run File usr local lib python3 10 site packages prefect backend flow run py line 94 in watch flow run flow run = flow run get latest File usr local lib python3 10 site packages prefect backend flow run py line 424 in get latest return self from flow run id File usr local lib python3 10 site packages prefect backend flow run py line 576 in from flow run id flow run data = cls query for flow run where= id eq flow run id File usr local lib python3 10 site packages prefect backend flow run py line 618 in query for flow run result = client graphql flow run query File usr local lib python3 10 site packages prefect client client py line 570 in graphql raise ClientError result errors prefect exceptions ClientError message request to failed reason connect ECONNREFUSED 10 10 11 202 3000 locations line 2 column 5 path flow run extensions code INTERNAL SERVER ERROR exception message request to failed reason connect ECONNREFUSED 10 10 11 202 3000 type system errno ECONNREFUSED code ECONNREFUSED 4 22 PM error message num 2 Failed to set task state with error ClientError message Unable to complete operation An internal API error occurred locations line 2 column 5 path set task run states extensions code INTERNAL SERVER ERROR exception message Unable to complete operation An internal API error occurred Traceback most recent call last File usr local lib python3 10 site packages prefect engine cloud task runner py line 91 in call runner target handlers state = self client set task run state File usr local lib python3 10 site packages prefect client client py line 1922 in set task run state result = self graphql File usr local lib python3 10 site packages prefect client client py line 570 in graphql raise ClientError result errors prefect exceptions ClientError message Unable to complete operation An internal API error occurred locations line 2 column 5 path set task run states extensions code INTERNAL SERVER ERROR exception message Unable to complete operation An internal API error occurred

Anna Geller · Answer

The internal server error is a generic message from an API call that failed and in this case it tells us that Prefect couldn t set task run states There can be many reasons for something like this to occur hard to say How do you run your Server Did this flow run ran on Kubernetes Did you use Dask It could be that a Kubernetes pod or a Dask worker client somehow died and therefore Prefect Server couldn t infer the task run state to be set Is any of this some long running job What is this flow doing What happens if you start a new flow run does it complete now i e a transient issue

Noam polak · Answer

Hey Anna We run prefect on Kubernetes Each parent flow creates 3 4 child flows and use their results` Each parent flow can lust 1 4 hours It only happens when we re trying to run multiple parent flow and it is resolved when I restart them

Khen Price · Answer

< Anna Geller> please notice there are 2 different errors < Noam polak> described we are on the same team one is related to what you mentioned failure to set the task state and another is regarding a refused connection ``` message request to failed reason connect ECONNREFUSED 10 10 11 202 3000 type system errno ECONNREFUSED code ECONNREFUSED ``` Does this give any clue on how to debug

Khen Price · Answer

Also would you suggest avoiding the approach of having flow of flows and using a more simple flow with tasks instead or should we expect flow of flows to be handled well by the system

Anna Geller · Answer

Flow of flows orchestration pattern is fully supported and works well so nothing speaks against that The connection refused maybe some networking issue on Kubernetes hard to say What Prefect version do you use on Server vs on your Kubernetes job template

Noam polak · Answer

Prefect Core Version 0 15 0

Noam polak · Answer

And our job templates for the flows are apiVersion batch v1

Anna Geller · Answer

I was only asking about Prefect version you use on your 1 Server 2 Agent 3 Flow e g set in `KubernetesRun` as image

Noam polak · Answer

server 2021 07 06 agent 0 15 0 flow 0 15 13

Anna Geller · Answer

I was asking about the versions because it could be caused by some Prefect version mismatch which causes that agent uses some API routes that are different in Server due to version mismatch The INTERNAL SERVER ERROR looks like a transient error to me perhaps Server was overloaded with too many requests at once and couldn t set task run states Regarding the ECONNREFUSED you can follow up on <https prefect community slack com archives CL09KU1K7 p1645611551384439 thread ts=1643660512 377699 amp cid=CL09KU1K7|this thread> it discussed the same error message potentially a similar issue