https://prefect.io
Join Slack
One last question for today... Mapping Tasks... is this still TODO on Prefect 2.0?
# ask-community
t
One last question for today... Mapping Tasks... is this still TODO on Prefect 2.0?
Or am I wanting Async tasks instead? Do you guys have documentation on those yet?
z
Both sync and async tasks can run concurrently. Here’s brief documentation on async tasks https://orion-docs.prefect.io/#async-concurrency
Mapping via a 
.map
operator is also on the roadmap and is currently being designed.
t
OK, we are liberal users of 
.map
and the 
DaskExecutor
in 1.2... would you recommend trying to rewrite as async, or should I wait until 
.map
is ready? Full adoption for us is waiting until it is out of beta, I am just trying to work ahead so waiting isn't a big deal.
z
I'd wait for the mapping operator if you want to spawn multiple mapped branches without blocking. Otherwise you can just use a for loop to iterate over the input value. The tasks will run concurrently without using async.
🙏 1
t
Hmmm.... i will probably revisit this tomorrow as my brain is done for the day.
OK, back on this with a fresh brain... which wants to say that the for loop is inherently syncronous so wouldn't it defeat the purpose of parallelizing with Dask? But I think I am going to try it out and see.
Getting this:
Copy code
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
so I must be doing something wrong. Asking in a new thread
z
The for loop is synchronous but all you’re doing is submitting task runs in it.
thor 1
That error is related to Dask multiprocessing, you need to call the flow within an if name == main block like that
t
Ah HA... typo in the if__name__ == "_main_" block
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by