Question about retries If I have a sequence of independent tasks, is it possible run them, and then rerun only those that failed ? but without waiting for the task that failed to rerun. only at the end of all the tasks get back to those that failed ?

From my understanding, you want to defer retries to the end. What is the use case you are retrying? To answer though, this can’t be done at the moment easily and not sure in what case this makes sense. The easiest way to kinda do this is have no retries, and then restart the flow from failure.

I have a flow that runs several independent ETLs, collecting and inserting data to some feature data-store tables in hadoop impala. some ETLs may run much faster than others, so if there is an issue with one of those that runs longer, I prefer the other ETLs not to have to wait for it to retry. another situation can be that one of those ETL have error, and again I prefer not to have to retry several times before running the other ETLs I do not want to run all the ETLs in parallel, since I do not want to overwhelm our cluster

So both of these are features only available in the Cloud where you can get Flow concurrency limits. For example, you can set a limit of 2 concurrent runs for flows with the label “x”. If 2 flows with label “x” are already running, the next flow runs with the same label will be queued until the previous ones finish. This can be done on the task level too. The second one is also possible with Cloud where you can set automations that change the state of your Flow if it has been running after a certain amount of time (mark it as Failed or Schedule again). In your case, I think the best thing you can do is control the concurrent flow runs using

wait_for_flow_run

task so that nothing gets started until that previous one is finished. I think you can also set upstream dependencies to prioritize shorter flows?

what happens if a flow get hang, and the next execution time arrives? will it run another instance of the flow, or wait for the existing flow to end ?

In Prefect Cloud, this is configurable since you can set a lot as failed after it’s been running for a set amount of time. The next execution will start a new flow run unless you tell it not to by enforcing FlowRun concurrency limits. You can also tell the next execution time to not start at all (cancel if late by X amount of time.)

I am looking at this

<https://code-maven.com/python-timeout>

You know Prefect has a timeout too on the task level right?

I am trying some imperfect way to terminate a flow after some time https://github.com/youdar/process_interupt_timer

Why can’t you do it with the

@task(timeout=…)

?

I am not using the cloud... does it work suppose to work for tasks when running Prefect on a VM ?

I think the confusion was that things can be timed out from inside task on both Server and Cloud. But only Cloud can stop something that is currently hanging (like something hangs. a worker died or something like that)