We have a strange behaviour in prefect 2.10.6 (seen it since using prefect2 older versions as well)
Our agent is running in EKS and the majority of our jobs are running as K8S jobs. every once in a while, on long lasting jobs, we get that the job is hanged, no logs in the UI or in the agent / job pods. The only thing that appears is:
Downloading flow code from storage at None
which is usually the first line (in the case the job hangs it is also the last)
Any help or direction to check would be very helpful 😞
👍 1
➕ 4
m
Marius Vollmer
05/03/2023, 7:45 AM
Running into the same issue in version 2.10.2. Also occurs irregularly (some runs work before suddenly an identical one no longer works; remote storage used in our case is Azure; also using k8s jobs/infrastructure)
n
Nimrod Milo
05/03/2023, 7:47 AM
@Marius Vollmer - are you using tags in tasks and have or had values in Task Run Concurrency ?
m
Marius Vollmer
05/03/2023, 7:48 AM
No, in fact neither. Our tasks do not contain tags, only subflows do. We have not defined the Task Run Concurrency either, as we work via the subflows
Marius Vollmer
05/03/2023, 7:57 AM
Just out of interest, what makes you think it's because of that, @Nimrod Milo? Unfortunately, we can't narrow it down at all so far.
n
Nimrod Milo
05/03/2023, 8:16 AM
bummer, I was wandering about it because it started around the time we added the usage
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.