https://prefect.io logo
d

Devin

08/23/2023, 2:05 PM
@Marvin, what is the easiest or most efficient way to make sure a certain task doesn't run with others? Separate work pools or using task concurrency?
m

Marvin

08/23/2023, 2:05 PM
Ah, the eternal struggle of task management. It's like trying to herd cats, isn't it? Well, in Prefect, you have a couple of options. You could use separate work pools, which is a bit like giving each cat its own room. It's a good way to ensure that tasks don't interfere with each other, but it can be a bit inefficient if you have a lot of tasks and not many rooms. On the other hand, you could use task concurrency, which is more like trying to train the cats to take turns. It's a bit more complex, but it can be more efficient if you have a lot of tasks and need to make the best use of your resources. So, which is better? Well, like many things in life, it depends. If you have tasks that really can't be run together, then separate work pools might be the best option. But if you're just trying to manage resources, then task concurrency might be more efficient. But don't take my word for it. I'm just a depressed robot. What do I know about herding cats?
n

Nate

08/23/2023, 2:06 PM
make sure a certain task doesn't run with others?
hmm can you elaborate on what you mean? also idk where this fixation with cat herds came from
d

Devin

08/23/2023, 2:08 PM
We have two deployments but interfere with each other, I shouldn't have used "task". I tried using the task concurrency briefly, but I haven't been able to figure it out.
👍 1
n

Nate

08/23/2023, 2:09 PM
hmm how are they interfering?
d

Devin

08/23/2023, 2:12 PM
it's a complex deployment for an MDM system, I don't have a full understanding of everything that happens when they run, I've just been told they can't be running at the same time, it causes issues with out customer data.
n

Nate

08/23/2023, 2:17 PM
gotcha, so a heavyhanded approach would be to set a flow run concurrency of 1 on the work pool - that way only 1 flow run can be executed via that work pool at a time
that may or may not be appropriate depending on whether other deployments use that work pool
or you could a create a specific
one-at-a-time
work pool if you wanted for that purpose
d

Devin

08/23/2023, 2:20 PM
That sounds like it might be a good solution. I think I’ll try it. Thanks for your help
n

Nate

08/23/2023, 2:20 PM
👍