Hi All - I am migrating from Prefect 1.0 to 2.0 and am specifically dealing with a mapped task parallelized using Dask. From the 2.0 docs it sounds like you need to call Task.submit() when using a task runner like Dask, but I am unclear how that is applied when using a map. Any guidance or an example would be appreciated. Thanks!

You mean using the function

map

of dask?

here's some pseudo code to illustrate:

@task
def my_task(arg):
    return arg

ActiveTaskRunner = DaskTaskRunner(address=dask_address)

# prefect
@flow(task_runner=ActiveTaskRunner)
def flow_test():
    task_futures = my_task.map(task_list)
    for future in task_futures:
        future.wait()
    # do more stuff with results of map?


flow_test()

ah I see. with

map

, you don’t need to use

.submit

to use task runner

oh perfect. Are there any additional tweaks needed to use the results from a map call? Running my code based on structure above I keep getting

TypeError: submit() got an unexpected keyword argument 'key'

Can you confirm if you have the TypeError without DaskTaskRunner?

Using ConcurrentTaskRunner it works

Can you upgrade to the latest version of Prefect and see if the error still persists?

i'm on 2.3.0 but I can try 2.3.1

@Ryan Peden Do you have any insights into this?

rebuilding a minimal environment to isolate this a bit further

Thanks - I'm running a few quick tests on my end as well

Here's full version info too: Version: 2.3.1 API version: 0.8.0 Python version: 3.10.6 Git commit: 1d485b1d Built: Thu, Sep 1, 2022 3:53 PM OS/Arch: linux/x86_64 Profile: default Server type: ephemeral Server: Database: sqlite SQLite version: 3.39.2

I get it too - I think it is a problem when using a

DaskTaskRunner

with Prefect 2.3.x. It seems to work fine with 2.2.x. I see we've got a high priority issue open for this in GitHub, so I expect we will have a fix published very soon. In the meantime, if you don't need any of the features or fixes added to 2.3.x, I'd recommend using Prefect 2.2.1.

Using 2.2.0 ( I did not find a 2.2.1) there is now an issue with the alembic package:

alembic.util.exc.CommandError: Can't locate revision identified by 'e757138e954a'

Ah - I think that is the Prefect database. 2.3 probably adds some non-backwards compatible changes. If you don't mind getting rid of any old flow runs you've recorded in it, running

rm -rf ~/.prefect

will take care of the issue you are seeing

that worked, thank you for all the help!