Hi everyone, has anyone tried using

asyncio.Event

to add inter-task dependency? I have asked @Marvin, but its example is incorrect.

It seems that there isn't a direct example or documentation available in Prefect on using

asyncio.Event

for inter-task dependencies. However, Prefect does have its own event system for emitting and handling events, which might be more suitable for managing dependencies within Prefect flows. You might want to consider using Prefect's built-in constructs like

wait_for

and

signals

to manage task dependencies. These are designed to work seamlessly within the Prefect ecosystem and might save you some unnecessary headaches. Here are some resources that might be useful: - Prefect Events Documentation - Prefect GitHub Source Code If you have any specific requirements or constraints, feel free to share, and I can help you explore other options.

Why won't prefect's own ways to create dependencies between tasks work for your use case? It's probably better practice not to have your own orchestrator trying to manage prefect's orchestrator.

Hi @Janet Carson, the issue seems to arise from running flows on different dasktaskrunners and tasks running on different nodes. Events cause hangs but perhaps it is because I was just trialling with a local prefect server. @Marvin could there be issues using

asyncio.Event

when running on a short lived prefect server?

Using

asyncio.Event

with Prefect, especially when dealing with

DaskTaskRunner

across different nodes, might indeed cause issues like hanging, particularly if you're using a short-lived local Prefect server. Here are some considerations: 1. DaskTaskRunner: Prefect's

DaskTaskRunner

submits tasks to the Dask distributed scheduler, and by default, creates a temporary Dask cluster for the flow run. If you're using a local server, ensure that the Dask cluster is correctly configured and accessible across all nodes. Dask Integration 2. Asyncio and Events: Prefect's event system is designed to emit events from flows, tasks, and other objects, but

asyncio.Event

handling can be tricky in distributed settings. Ensure that your event handling logic is robust against network latencies or interruptions. 3. Common 3.x Gotchas: Prefect 3.x has replaced some core components, like agents with workers, which might affect how tasks are scheduled and run. Ensure your setup aligns with these changes. Prefect 3.x Gotchas 4. GitHub Issues: There don't appear to be any open issues related directly to

asyncio.Event

hanging with

DaskTaskRunner

, but it's always wise to keep an eye on the repository for any updates. If you're encountering persistent issues, consider running your setup on a more stable Prefect server or cloud-hosted Dask cluster to rule out local server constraints.

Yes - mine was a higher level question - why are you using asyncio events at all rather than the orchestration tools available within prefect itself?

Hi @Janet Carson, what orchestration tools in prefect would allow concurrent subflows running on different slurm clusters? When I asked and looked at the documentation, it seemed like I would need asyncio. Added to the complexity is that I have a subflow running a task that needs to happen before certain sets of tasks running in another subflow (because they need a different slurm cluster) begin.

So, if the tasks are in completely separate flows, I think I see the problem. The question for me becomes "but then is prefect the solution" for this application architecture, because prefect is trying to manage which flows/tasks are dependent on each other and coordinate them and then you have this extra stuff added to also orchestrate and at some point they may come into conflict with one another. It's bound to cause headaches at some point if you have two orchestrators at the same time.

I would say

prefect

isn’t up to it but then, I haven’t found a workflow manager that is up to it and prefect seemed like a good start point

If you are going to use an orchestrator, you have to let it orchestrate -- you can't have some of the orchestration in your own code and some in the orchestrator. That may mean some refactoring of how you presently do things.

Hi Janet, it would be great to arrange a call because I am trying to let the orchestrator orchestrate but it isn’t up to the task and missing functionality. I will say that I wrote this workflow from scratch with prefect in mind so there is no refactoring to do. Or more precisely, I was always looking for how to use prefect to generate intertask dependencies between tasks running using different dask task managers with actual processes running on different compute nodes.

I'm not a prefect employee, I'm just a community member. Since I only use prefect, I wouldn't be the best consultant to hire to find an orchestrator that has the functionality you are looking for.

Ah, cool thanks @Janet Carson