Hi all - was looking for advice on how best to set...
# prefect-clouda
Ameya
12/18/2023, 3:55 PMHi all - was looking for advice on how best to set up work queues/pools/concurrency limits (or something else) to achieve the following:
Say we have ingest and egress jobs for specific data sets, that can occur at any time when people kick them off. I would want the ingest/egress jobs to be run in a single queue per data set, so that the jobs don't overlap with one another. I think this would work with a separate work queue by data set with a concurrency limit of 1. However, the problem is that we'll have hundreds of datasets, that are constantly evolving, so that would be a pain to set up. Is there a good way to achieve this?
b
Bianca Hoch
12/19/2023, 3:13 PMHi Ameya 👋 . Just to confirm- you only want one ingest/egress flow running at a time for a given dataset?
Bianca Hoch
12/19/2023, 3:15 PMJust spitballing here, but maybe instead of having a queue-per-dataset, you could have a task concurrency tag per dataset instead? with a limit of 1?
Bianca Hoch
12/19/2023, 3:16 PMBianca Hoch
12/19/2023, 3:24 PMIn theory you could do the same with work queues using create_work_queue. You could programmatically create a new queue per data set, and set a concurrency limit of 1.
Bianca Hoch
12/19/2023, 3:25 PMHaving hundreds of unique task tags may be more practical than hundreds of queues 🤔
a
Ameya
12/19/2023, 3:30 PMyes, confirming only 1 ingress/egress job should be running at a time for a given dataset. the jobs are flows instead of tasks - can the tag concurrency limit be applied to full flows or only tasks?
b
Bianca Hoch
12/19/2023, 3:34 PMTask tags would be specific to tasks
Bianca Hoch
12/19/2023, 3:39 PMSo it'd require a bit of refactoring. My thinking is that an Ingress flow kicks off an ingress task with a concurrency limit of 1.
Bianca Hoch
12/19/2023, 4:05 PMhopefully a qq, but do the ingress and egress jobs run one after the other? ie: Once the ingress flow completes it kicks off a egress flow?
a
Ameya
12/19/2023, 4:20 PMhmm ok that wouldn't work unfortunately since the flows are semi-complicated with many tasks in them.
Ameya
12/19/2023, 4:21 PMingress and egress jobs don't run in the same flow or at the same time - someone could kick off an extract at any point, regardless of ingress schedule
Ameya
12/26/2023, 3:31 PM@Bianca Hoch - do you know if there is a limit to the number of work queues you can have? Also, do you know when work queues are going to be added to the prefect terraform provider?
b
Bianca Hoch
12/26/2023, 10:29 PMHi Ameya, I don't believe there is a hard limit to the number of work queues that are allowed.
As for when work queues are going to be added to the terraform provider, I don't think there is an ETA. If you don't mind, could you create a feature request for our engineers?
a
Ameya
12/26/2023, 10:39 PMthanks - submitted here! https://github.com/PrefectHQ/terraform-provider-prefect/issues/131
3 Views