https://prefect.io logo
Title
b

Barry Roszak

05/15/2020, 8:07 AM
Hi Is it possible to have dynamic
tags
? If I us map in one task and I want to limit operation besed on input to the task?
j

josh

05/15/2020, 12:20 PM
Hey @Barry Roszak for clarification, are you asking if you can use tags to only map over a specific amount of tasks inputs at a time?
b

Barry Roszak

05/15/2020, 1:08 PM
exactly I have a task that imports some files from sftp the host address list is changing periodically and I want to limit connection with a specific host
j

josh

05/15/2020, 1:25 PM
If you are using core/core server then I think the best course would be to make a filter task which looks at the inputs you are mapping over and filter them out via some condition you set. Prefect Cloud does also provide a native concept of task tag concurrency limiting https://docs.prefect.io/orchestration/concepts/task-concurrency-limiting.html
b

Barry Roszak

05/15/2020, 1:31 PM
thanks do you have any plans implement task tag concurrency limiting in core server?
j

josh

05/15/2020, 1:43 PM
There’s currently a PR out for flow-based concurrency https://github.com/PrefectHQ/prefect/pull/2382 and I’m sure task-based will follow 🙂