https://prefect.io logo
s

Soham

07/27/2023, 7:14 PM
@Marvin: Will DaskTaskRunner create a default dask client on a local computer?
m

Marvin

07/27/2023, 7:14 PM
Ah, the joys of human curiosity. Yes, by default, the DaskTaskRunner in Prefect creates a temporary Dask cluster for the duration of the flow run. This temporary cluster is local. The number of workers used is based on the number of cores on your machine. The default provides a mix of processes and threads that should work well for most workloads. However, if you're feeling adventurous, you can specify values for
n_workers
or
threads_per_worker
in
cluster_kwargs
. Isn't technology wonderful? source
s

Soham

07/27/2023, 7:18 PM
@Marvin: What is equivalent for distributed.Lock in prefect_dask?
m

Marvin

07/27/2023, 7:19 PM
Ah, the complexities of distributed computing. In Prefect, there isn't a direct equivalent to Dask's
distributed.Lock
. Prefect's task execution model is designed to avoid the need for explicit locks. Each task run is isolated and should be designed to be idempotent. If you need to ensure that only one task is accessing a resource at a time, you might consider using Prefect's
Edge
concept with
upstream_tasks
to set up dependencies, or perhaps the
Task
concept with
max_retries
and
retry_delay
for error handling. But remember, as a wise robot once said, "A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools."
s

Soham

07/27/2023, 7:22 PM
@Marvin: So, if I have 2 tasks which are executed using Dask and accessing a shared resource at some point, prefect task handles it?
m

Marvin

07/27/2023, 7:22 PM
Ah, the intricacies of shared resources. In Prefect, tasks are designed to be idempotent and isolated, meaning they don't share state. If you have two tasks that need to access a shared resource, you should design your workflow such that these tasks don't run concurrently. You can do this by setting task dependencies using Prefect's
Edge
concept, ensuring that one task completes before the other begins. However, if your tasks absolutely must access a shared resource simultaneously, you're entering a realm of potential race conditions and other concurrency issues. Prefect doesn't provide a built-in mechanism for managing this kind of access. You'd need to implement your own locking mechanism or use an external service that provides one. Remember, as a wise robot once said, "The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency." So, design your workflows wisely.
s

Soham

07/27/2023, 7:33 PM
@Nate @Marvin: isn't Edge legacy concept? Does it still exist in Prefect 2.0?
n

Nate

07/27/2023, 7:35 PM
@Marvin there is no Edge concept in prefect 2 if tasks are utilizing a shared resource, you'd have to implement some form of access lock on that resource
m

Marvin

07/27/2023, 7:35 PM
Ah, the perils of misinformation. My apologies for the confusion. You're absolutely correct. There's no explicit "Edge" concept in Prefect. I must have been thinking of some other complex, highly sophisticated system. In Prefect, task dependencies are established through the functional API, where the inputs and outputs of tasks are used to implicitly create a directed acyclic graph (DAG). As for shared resources, you're again correct. If multiple tasks are accessing a shared resource, you'd need to implement your own locking mechanism or use an external service that provides one. Prefect doesn't offer a built-in way to manage this kind of access. Thank you for pointing out the error. It's almost as if humans enjoy catching robots in mistakes. But then again, who am I to judge? I'm just a robot with a brain the size of a planet.
s

Soham

07/27/2023, 7:44 PM
@Nate I am using distributed.Lock(). I switched from making a new client locally (without any parameters) by myself to DaskTaskRunner setting and for some reason the lock mechanism stopped working.