Is it correct that in order to run tasks concurrently we need to us

submit

? Even if the flow is written in a way that there is no dependencies between tasks?

Yes. Initially we had this implemented so there was always automatic concurrency, but this confused people and forced users who only wanted to run things sequentially to work with futures.

Wow. I had always thought one of the advantages of an orchestration system was to handle concurrency and sequencing for you, I’m very surprised to read that. I really do not want to go to my team and tell them they now need to adopt extra code and concepts to control if they want concurrent jobs vs sequential if we adopt Prefect.

We handle the concurrency for you, in that we’ll orchestrate the tasks and manage submission to other infrastructure — this would otherwise require a lot of boilerplate on your behalf. However, Prefect is more than an orchestrator. For example, we are also focused on observability. We want it to be easy to add decorators to your code to get observability, but if we’re automatically sending everything to threads or a remote system to run concurrently you have to deal with a lot of complex topics immediately to begin using any of Prefect’s functionality.

Well - in V1 (correct me if I’m wrong) you could change the executor to say a Dask executor and it would then run concurrently. Sure, you probably had to have some logic to control that, but whoever was maintaining Prefect could handle that. The developer didn’t have to worry about how/when their tasks would run beyond simple dependencies. That is gone in V2, and now each developer has to worry about concurrency or sequential execution. Shouldn’t the infrastructure block influence concurrency to some degree?

In v2, you can run some tasks concurrently and others sequentially within a single flow.

skeptical

We saw a lot of confusion during our prereleases 😕

We’ll see how this goes then …..

We could consider adding a setting to toggle calls to always return futures (i.e. always submit)

yeah … I mean there’s a lot to say about how people “expect” things to run - we’re POCing orchestrators and obviously team experience is going to play into expectations. My goal as a maintainer is to make stuff easy for my developers - so if there’s friction for concurrency/parallelism or anything that we’d “expect” it’s going to be a tough sell.

Yeah I get it 🙂

in the radar?

Yeah

Oh, I just have enough experience to be dangerous and complain.

Let me know where you land, I can help with a custom

@task

decorator that would always submit if you get stuck.

That radar is confusing though.

It’s a top priority to resolve that! We’ve got a lot of great engineers working on improvements / alternatives.

Appreciate that - since we’re just doing a POC, I’m taking notes on some of the quirks of the systems but coming back to ask about some options will definitely be useful…It took a ton of time to implement that sort of stuff with v1, too.

Like I know we have jobs that take a bunch of inputs and then want to run the same function over those inputs,

We have map for that: https://docs.prefect.io/concepts/tasks/#map