Hi, i have a task function which retries multiple times to check if a file is present and if successful starts the downstream task. This same task has different parameters going into it, the ones in orange color are working correctly (going into retries) but few of them are in yellow color (going in submitted state) and causing downstream task to fail. What is this submitted task? And why is it happening ?

It looks like you are using a DaskExecutor and the yellow Submitted states indicate that a task run has been submitted to Dask for execution. 1. Are the yellow tasks stuck and stay in Submitted state forever? 2. How did you define this polling mechanism - is it part of a mapped task? What is your max retries on that task? Btw, not sure if this is helpful since you seem to have already implemented this, but here is a Discourse topic that describes different ways of approaching similar use cases

1. Yes the yellow task is stuck in submitted state. 2. No, polling mechanism for file check is a basic shell command running through ShellTask. It has 30 retries with 15 minute interval. The same task is running fine for the orange ones but randomly failing for the yellow ones.

Can you share the flow or a simplified version so that we can try to reproduce?