@Marvin can you give me an example of how i can parallelize my flow using dasktaskrunner? i keep running into a memory limit on my docker image which im running it on

To run your flow with parallel execution using the DaskTaskRunner, you can specify it in the flow decorator. Here are some helpful links and a basic example: Docs Excerpt: DaskTaskRunner can spin up a local Dask cluster or connect to an environment you've set up. You can execute tasks asynchronously using the

.submit

method. Read more from these guides: - Dask and Ray Task Runners - Task Runners - Global Concurrency Limits Example Code: Here's a simplified example of how you can use DaskTaskRunner to parallelize your tasks.

from prefect import flow, task
from prefect_dask import DaskTaskRunner

@task
def add(x, y):
    return x + y

@flow(task_runner=DaskTaskRunner)
def my_parallel_flow():
    results = []
    for i in range(10):
        results.append(add.submit(i, i+1))
    return results

if __name__ == "__main__":
    my_parallel_flow()

By configuring

DaskTaskRunner

, your tasks will be executed in parallel, which should help manage memory usage with Docker. You can further explore the provided links for advanced configurations and optimizations based on your needs.