is there a way to limit task submissions from a ma...
# ask-communitya
Andy Dyer
12/21/2023, 7:03 PMis there a way to limit task submissions from a map
some_task.map(ten_thousand_item_list)
is there a way to stop 10,000 tasks from being submitted at once?
✅ 1
k
Kevin Grismore
12/21/2023, 7:04 PMyou can use tags to set task run concurrency limits
a
Andy Dyer
12/21/2023, 7:05 PMyeah that stop them from executing
Andy Dyer
12/21/2023, 7:05 PMbut i dont want to have the actual tasks submit because that wont contribute to orchestration API limit
k
Kevin Grismore
12/21/2023, 7:05 PMah that's a good point
a
Andy Dyer
12/21/2023, 7:06 PMobviously can batch up the work into N equal chunks then map them
Andy Dyer
12/21/2023, 7:06 PMbut not ideal
k
Kevin Grismore
12/21/2023, 7:06 PMyeah I was going to suggest that but it's not the most satisfying answer
a
Andy Dyer
12/21/2023, 7:09 PMwould be ideal if map could take a kwarg of like "batch_size" or something similar
Andy Dyer
12/21/2023, 7:12 PMany idea or should i just go with the chunking?
k
Kevin Grismore
12/21/2023, 7:13 PMyou could adjust your client retry settings https://docs.prefect.io/latest/api-ref/prefect/settings/?h=settings#prefect.settings.PREFECT_CLIENT_MAX_RETRIES
Kevin Grismore
12/21/2023, 7:14 PMand then try to align the retries on 429s with the runtime of your mapped tasks
Kevin Grismore
12/21/2023, 7:14 PMbut that's pretty imprecise too I think
a
Andy Dyer
12/21/2023, 7:14 PMyeah not ideal
Andy Dyer
12/21/2023, 7:15 PMI think im just going to do 1000 at a time and live with it, but this would be ideal as an API config on map task
Andy Dyer
12/21/2023, 7:15 PMnot sure the technical feasibility of it but yeah
k
Kevin Grismore
12/21/2023, 7:25 PMyou can rate limit the submission of tasks using our global concurrency tools
a
Andy Dyer
12/21/2023, 7:36 PMand i assume in this context the slow_my_flow references a tag
k
Kevin Grismore
12/21/2023, 7:40 PMa global concurrency limit
Kevin Grismore
12/21/2023, 7:41 PMsettings -> concurrency -> global tab
a
Andy Dyer
12/21/2023, 7:41 PMappreciate it ill give it a whirl now :^)
🙌 1
Andy Dyer
12/21/2023, 7:43 PM Copy code
@flow
def _check_file_path_flow():
mapped_files = get_already_mapped_files()
for record in mapped_files:
rate_limit("file_api", occupy=1)
updated_path = get_updated_path(record)
update_path_and_name(updated_path)Andy Dyer
12/21/2023, 7:44 PMlooks like its submitting sequentially still, granted i just cursory glanced for these values on the link you sent
k
Kevin Grismore
12/21/2023, 7:44 PMget_updated_path.submit(record)a
Andy Dyer
12/21/2023, 7:45 PMoh lol right
k
Kevin Grismore
12/21/2023, 7:45 PMyou're good lol
a
Andy Dyer
12/21/2023, 7:45 PMnow its ripping
🙌 1
Andy Dyer
12/21/2023, 7:45 PMthanks Kevin
k
Kevin Grismore
12/21/2023, 7:45 PMyou got it
a
Andy Dyer
12/21/2023, 7:46 PMfwiw itd be nice to have a tooltip in the UI for
Andy Dyer
12/21/2023, 7:46 PMor a link to docs
k
Kevin Grismore
12/21/2023, 7:47 PMthere's a link to docs if you haven't created a concurrency limit yet
a
Andy Dyer
12/21/2023, 7:47 PM👍
8 Views