https://prefect.io
Join Slack
Random quick question I hope, is it possible to run a task with a .submit and .map at the same time
# ask-community
n
Random quick question I hope, is it possible to run a task with a .submit and .map at the same time
z
Yep!
like
Copy code
foo.submit(1)
foo.map([2, 3])
?
n
Like foo.sumbit(1).map(2,3)?
z
That doesn’t really make sense. What are you trying to do?
n
So I'm trying to run a mapped task and not have tasks after it wait for the mapped task to finishing mapping before they start, I'll put some code up in a bit.
cta_load_list : PrefectFuture = get_security_master_staging_cta_load_list.submit()
cta_load_results = run_stored_procedure.map(
sql_util=unmapped(staging_sql_util), procedure_name=cta_load_list
)
security_master_side_load_list: PrefectFuture = (
get_security_master_staging_dw_side_load_list.submit()
side_load_results = run_stored_procedure.map(
sql_util=unmapped(dw_sql_util), procedure_name=security_master_side_load_list
)
So the 3rd list of get_security_master_side_load_list doesn't start until the mapping has finished mapping out. Would it be better if I put a waitfor on the 3rd line?
z
Mapping requires us to resolve the length of the input future in order to return the full list of mapped futures
n
Ok, I think I get it. So a map is basically the same as a submit, but there's like a couple of milliseconds/seconds after the map before the next task on the list can start?
Thanks for going through this with me by the way, these are the types of questions I am getting when trying to introduce this to the company, which usually uses C# for this sort of thing.
z
👍 map does a submit behind the scenes for each item in the iterable(s) it receives
It’s equivalent to
Copy code
for i in items:
   task.submit(i)

# same as task.map(items)
(with some more nice sugar and such)
If 
items
is a future we need to resolve the future to data (wait for it to complete) first so we can make those submissions
n
Gotcha, makes a lot of sense to me, thanks!
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by