I can’t understand why I have a flow where task run more than one time. For example in the image the...

Giovanni Giacco

11/15/2021, 6:28 PM

I can’t understand why I have a flow where task run more than one time. For example in the image there is a Parameter Task. Any suggestion?

Kevin Kho

11/15/2021, 6:29 PM

Hey @Giovanni Giacco, did you register a flow with

flow.run()

inside?

Giovanni Giacco

11/15/2021, 6:30 PM

I registered it to Prefect Cloud with flow.register() and executing it from the GUI

Kevin Kho

11/15/2021, 6:30 PM

Could you share your flow code?

Giovanni Giacco

11/15/2021, 6:31 PM

It’s a very long flow. I’m trying to create a minimum example.

Giovanni Giacco

11/15/2021, 6:43 PM

Anyway I have not registered a flow with

flow.run()

inside and I noticed that with LocalExecutor it doesn’t happen but with Dask Executor and KubernetesRun yes.

Kevin Kho

11/15/2021, 6:46 PM

Ah I see. Are you on Server?

Giovanni Giacco

11/15/2021, 6:48 PM

I don’t know if I understood the question. I’m using Prefect Cloud.

Kevin Kho

11/15/2021, 6:49 PM

Ah sorry. Yeah that was my question if you were on Prefect Cloud or Prefect Server.

Kevin Kho

11/15/2021, 6:53 PM

So this can happen if Prefect submits tasks to Dask and then the Dask worker dies (can happen from memory issues). Dask itself has a failure handling mechanism that will restart the worker and restart the tasks on the worker. So what happens is Dask will attempt to retry tasks that have succeeded before already. For example, Prefect submitted tasks A, B and C to Dask and then run in the order A -> B -> C. If A and B succeeded, but then the Dask worker dies during C, Dask will try to start A and B again when it spins up a new worker. To avoid this, Prefect has Version Locking which will prevent those tasks from running again. If this is enabled on your Flow, then it won’t run those tasks again even if attempted

Giovanni Giacco

11/15/2021, 7:04 PM

I didn’t know about Version Locking. Thank!🙏 Just to understand, is this feature available only for Prefect Cloud?

Kevin Kho

11/15/2021, 7:05 PM

Yes exactly that’s why I was asking if you were on Server

Giovanni Giacco

11/15/2021, 7:23 PM

I can’t activate the version locking from the GUI. The message is: “your team doesn’t have access to version locking”. Is a premium feature?

Kevin Kho

11/15/2021, 7:28 PM

It is standard tier and above

👍 1

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.