https://prefect.io logo
Title
n

Nimrod Milo

05/02/2023, 7:46 AM
We have a strange behaviour in prefect 2.10.6 (seen it since using prefect2 older versions as well) Our agent is running in EKS and the majority of our jobs are running as K8S jobs. every once in a while, on long lasting jobs, we get that the job is hanged, no logs in the UI or in the agent / job pods. The only thing that appears is:
Downloading flow code from storage at None
which is usually the first line (in the case the job hangs it is also the last) Any help or direction to check would be very helpful 😞
👍 1
3
m

Marius Vollmer

05/03/2023, 7:45 AM
Running into the same issue in version 2.10.2. Also occurs irregularly (some runs work before suddenly an identical one no longer works; remote storage used in our case is Azure; also using k8s jobs/infrastructure)
n

Nimrod Milo

05/03/2023, 7:47 AM
@Marius Vollmer - are you using tags in tasks and have or had values in Task Run Concurrency ?
m

Marius Vollmer

05/03/2023, 7:48 AM
No, in fact neither. Our tasks do not contain tags, only subflows do. We have not defined the Task Run Concurrency either, as we work via the subflows
Just out of interest, what makes you think it's because of that, @Nimrod Milo? Unfortunately, we can't narrow it down at all so far.
n

Nimrod Milo

05/03/2023, 8:16 AM
bummer, I was wandering about it because it started around the time we added the usage