Hello all. I have 2 tasks in a flow, execute_a and...
# prefect-community
j
Hello all. I have 2 tasks in a flow, execute_a and execute_b. The task execute_a is using raise LOOP to make successive calls to an API and will return a string when it finishes. When I simulate an error (eg. API is down), I notice that execute_a is set to Retry state and execute_b is also started and set to Pending state (I was expecting execute_b to not even start). At this point the API will become available again and eventually both tasks will finish with success. My question here is, how can we tell Prefect to only start execute_b if execute_a is finished?
j
Hi Martins, that sounds like a possible bug - would you be willing to file an issue https://github.com/PrefectHQ/prefect/issues with a reproducible example so we can investigate further?
Also, for future posts, please make use of slack threads for follow-up comments. This helps keep things organized and easier to read. Thanks!
j
Copy code
with Flow("myflow") as flow:
  result_a = execute_a()
  result_b = execute_b(result_a)
I deleted the new thread and posted the code example here
I was wondering if this is just how Prefect behaves or if there is a way to prevent execute_b from starting
j
execute_b
should only run when
execute_a
has reached a finished state. If you are seeing differently, this sounds like a bug.
j
In practice I will get the expected result when the execute_a resumes
it is just in the logs that one can see that execute_b actually started and was put to Pending
can it be that the first iteration of the loop returns a finished state, hence the behaviour?
j
Does the
execute_b
task actually start running too early, or is it just the logs that are confusing? If the flow run still progresses as expected (with
execute_a
retrying, eventually succeeding, then
execute_b
running) then this isn't necessarily a bug.
j
...
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=‘myhost.com’, port=443): Read timed out. (read timeout=10) [2021-01-20 175002+0100] INFO - prefect.TaskRunner | Task ‘execute_a’: Finished task run for task with final state: ‘Retrying’ [2021-01-20 175002+0100] INFO - prefect.TaskRunner | Task ‘execute_b’: Starting task run... [2021-01-20 175002+0100] INFO - prefect.TaskRunner | Task ‘execute_b’: Finished task run for task with final state: ‘Pending’ [2021-01-20 175002+0100] INFO - prefect.FlowRunner | Flow run RUNNING: terminal tasks are incomplete. [2021-01-20 175002+0100] INFO - prefect.Myflow | Waiting for next available Task run at 2021-01-20T165102.371101+00:00
j
How long is your retry set for?
j
this is what happens, right after the exception happens (because the API is down) execute_a is set to Retrying and execute_b starts immediately
retry is set for 60 seconds delay
j
Hmmm, ok. I'd call this a bug (or at least something that could be better). Would you mind opening an issue with a reproducible example of the above? You can leave the body of `execute_a`/`execute_b` blank, just want the flow/task level settings to reproduce.
j
ok I will look into that, most probably only doing it tomorrow though
👍 1
I am very new to Prefect, thought it would be something I am missing