Seth Goodman
09/02/2022, 2:25 PMKhuyen Tran
09/02/2022, 2:42 PMmap
of dask?Seth Goodman
09/02/2022, 2:46 PM@task
def my_task(arg):
return arg
ActiveTaskRunner = DaskTaskRunner(address=dask_address)
# prefect
@flow(task_runner=ActiveTaskRunner)
def flow_test():
task_futures = my_task.map(task_list)
for future in task_futures:
future.wait()
# do more stuff with results of map?
flow_test()
Khuyen Tran
09/02/2022, 2:47 PMmap
, you don’t need to use .submit
to use task runnerSeth Goodman
09/02/2022, 2:48 PMTypeError: submit() got an unexpected keyword argument 'key'
@task
def map_print(x):
print(x)
@flow(task_runner=DaskTaskRunner())
def my_flow():
futures = map_print.map([1,2,3,4,5])
for future in futures:
future.wait()
# print(futures)
my_flow()
Khuyen Tran
09/02/2022, 2:57 PMSeth Goodman
09/02/2022, 2:58 PMKhuyen Tran
09/02/2022, 3:04 PMSeth Goodman
09/02/2022, 3:06 PMKhuyen Tran
09/02/2022, 3:09 PMSeth Goodman
09/02/2022, 3:17 PMRyan Peden
09/02/2022, 3:25 PMSeth Goodman
09/02/2022, 3:27 PMRyan Peden
09/02/2022, 3:37 PMDaskTaskRunner
with Prefect 2.3.x. It seems to work fine with 2.2.x. I see we've got a high priority issue open for this in GitHub, so I expect we will have a fix published very soon.
In the meantime, if you don't need any of the features or fixes added to 2.3.x, I'd recommend using Prefect 2.2.1.Seth Goodman
09/02/2022, 3:48 PMalembic.util.exc.CommandError: Can't locate revision identified by 'e757138e954a'
Ryan Peden
09/02/2022, 3:53 PMrm -rf ~/.prefect
will take care of the issue you are seeingSeth Goodman
09/02/2022, 3:55 PM