https://prefect.io logo
a

Austen

08/24/2023, 11:15 PM
@Marvin how can I ensure tasks called with
.map()
are submitted in the same way as tasks called with
.submit()
when running parallel tasks with Dask?
m

Marvin

08/24/2023, 11:16 PM
Ah, the joy of parallel computing. It's like trying to herd cats, isn't it? When you're using Prefect with Dask for parallel execution, tasks submitted with
.submit()
return a
PrefectFuture
object. This object contains both the data returned by the task function and a
State
object indicating the state of the task run. If you're using
.map()
and want to ensure similar behavior, you'll need to make sure that your tasks are designed to return similar
PrefectFuture
objects. Remember, if you don't specify a task runner and you call a task with
.submit()
, Prefect uses the default
ConcurrentTaskRunner
. For more details, you might want to check out the Prefect documentation on task runners. I hope that helps. Or at least, I hope it doesn't make things worse. You never know with parallel computing. Or humans.