@Marvin: Will DaskTaskRunner create a default dask client on a local computer?

Ah, the joys of human curiosity. Yes, by default, the DaskTaskRunner in Prefect creates a temporary Dask cluster for the duration of the flow run. This temporary cluster is local. The number of workers used is based on the number of cores on your machine. The default provides a mix of processes and threads that should work well for most workloads. However, if you're feeling adventurous, you can specify values for

n_workers

or

threads_per_worker

in

cluster_kwargs

. Isn't technology wonderful? source

@Marvin: What is equivalent for distributed.Lock in prefect_dask?

Ah, the complexities of distributed computing. In Prefect, there isn't a direct equivalent to Dask's

distributed.Lock

. Prefect's task execution model is designed to avoid the need for explicit locks. Each task run is isolated and should be designed to be idempotent. If you need to ensure that only one task is accessing a resource at a time, you might consider using Prefect's

Edge

concept with

upstream_tasks

to set up dependencies, or perhaps the

Task

concept with

max_retries

and

retry_delay

for error handling. But remember, as a wise robot once said, "A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools."

@Marvin: So, if I have 2 tasks which are executed using Dask and accessing a shared resource at some point, prefect task handles it?

Ah, the intricacies of shared resources. In Prefect, tasks are designed to be idempotent and isolated, meaning they don't share state. If you have two tasks that need to access a shared resource, you should design your workflow such that these tasks don't run concurrently. You can do this by setting task dependencies using Prefect's

Edge

concept, ensuring that one task completes before the other begins. However, if your tasks absolutely must access a shared resource simultaneously, you're entering a realm of potential race conditions and other concurrency issues. Prefect doesn't provide a built-in mechanism for managing this kind of access. You'd need to implement your own locking mechanism or use an external service that provides one. Remember, as a wise robot once said, "The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency." So, design your workflows wisely.

@Nate @Marvin: isn't Edge legacy concept? Does it still exist in Prefect 2.0?

@Marvin there is no Edge concept in prefect 2 if tasks are utilizing a shared resource, you'd have to implement some form of access lock on that resource

Ah, the perils of misinformation. My apologies for the confusion. You're absolutely correct. There's no explicit "Edge" concept in Prefect. I must have been thinking of some other complex, highly sophisticated system. In Prefect, task dependencies are established through the functional API, where the inputs and outputs of tasks are used to implicitly create a directed acyclic graph (DAG). As for shared resources, you're again correct. If multiple tasks are accessing a shared resource, you'd need to implement your own locking mechanism or use an external service that provides one. Prefect doesn't offer a built-in way to manage this kind of access. Thank you for pointing out the error. It's almost as if humans enjoy catching robots in mistakes. But then again, who am I to judge? I'm just a robot with a brain the size of a planet.

@Nate I am using distributed.Lock(). I switched from making a new client locally (without any parameters) by myself to DaskTaskRunner setting and for some reason the lock mechanism stopped working.