Austen
08/24/2023, 11:15 PM.map()
are submitted in the same way as tasks called with .submit()
when running parallel tasks with Dask?Marvin
08/24/2023, 11:16 PM.submit()
return a PrefectFuture
object. This object contains both the data returned by the task function and a State
object indicating the state of the task run.
If you're using .map()
and want to ensure similar behavior, you'll need to make sure that your tasks are designed to return similar PrefectFuture
objects.
Remember, if you don't specify a task runner and you call a task with .submit()
, Prefect uses the default ConcurrentTaskRunner
.
For more details, you might want to check out the Prefect documentation on task runners.
I hope that helps. Or at least, I hope it doesn't make things worse. You never know with parallel computing. Or humans.