https://prefect.io logo
Title
n

Nathan Low

04/26/2023, 4:45 PM
Random quick question I hope, is it possible to run a task with a .submit and .map at the same time
z

Zanie

04/26/2023, 5:05 PM
Yep!
like
foo.submit(1)
foo.map([2, 3])
?
n

Nathan Low

04/26/2023, 5:59 PM
Like foo.sumbit(1).map(2,3)?
z

Zanie

04/26/2023, 6:16 PM
That doesn’t really make sense. What are you trying to do?
n

Nathan Low

04/26/2023, 6:17 PM
So I'm trying to run a mapped task and not have tasks after it wait for the mapped task to finishing mapping before they start, I'll put some code up in a bit.
cta_load_list : PrefectFuture = get_security_master_staging_cta_load_list.submit()
cta_load_results = run_stored_procedure.map(
sql_util=unmapped(staging_sql_util), procedure_name=cta_load_list
)
security_master_side_load_list: PrefectFuture = (
get_security_master_staging_dw_side_load_list.submit()
side_load_results = run_stored_procedure.map(
sql_util=unmapped(dw_sql_util), procedure_name=security_master_side_load_list
)
So the 3rd list of get_security_master_side_load_list doesn't start until the mapping has finished mapping out. Would it be better if I put a waitfor on the 3rd line?
z

Zanie

04/26/2023, 6:24 PM
Mapping requires us to resolve the length of the input future in order to return the full list of mapped futures
n

Nathan Low

04/26/2023, 6:47 PM
Ok, I think I get it. So a map is basically the same as a submit, but there's like a couple of milliseconds/seconds after the map before the next task on the list can start?
Thanks for going through this with me by the way, these are the types of questions I am getting when trying to introduce this to the company, which usually uses C# for this sort of thing.
z

Zanie

04/26/2023, 6:49 PM
👍 map does a submit behind the scenes for each item in the iterable(s) it receives
It’s equivalent to
for i in items:
   task.submit(i)

# same as task.map(items)
(with some more nice sugar and such)
If
items
is a future we need to resolve the future to data (wait for it to complete) first so we can make those submissions
n

Nathan Low

04/26/2023, 7:02 PM
Gotcha, makes a lot of sense to me, thanks!