Anyone else had the problem of mapped tasks succeeding with every sub-task but staying in a

mapped

state?

Hi @matta - that’s expected behavior, since Mapped is a specialized inheritor of the Success state

@nicholas Ah, nice! So, I'm having a larger problem of flows not terminating after everything succeeds. I thought that might have been it.

I’ve seen that before - are you running on dask?

Yup!

Ah so at the end of the run, the flow runner process gathers all task states (including results!), and sometimes that can either hang the dask scheduler or the process runs out of memory. If you could check the memory / CPU of that process and provide more that could resolve it. Longer term we would like to improve performance around things like this but we don’t have a concrete plan for it yet

@Chris White Does it make a difference if I'm using threads?

oh whoops I forgot my phone and computer were different accounts lol

Hm no if you aren’t using an external cluster then I’m not sure that my story makes sense 🧐

Maybe putting a memory limit back might help?

No but you might install an editable install of prefect and add some breakpoints or logs in the flow runner to see where it freezes; alternatively there are some interesting Python packages that let you analyze another Python process but I’m very rusty on the details there

@Chris White Is there a guide on using the Dask UI while using Prefect on a remote server? Or do you mean like using the Dask UI in Jupyter?

Interesting - and no guide for that, while the flow is running you should be able to port forward to the server on whatever port the dashboard is exposed on

@Chris White Is there a way to do that if I don't have SSH access to the Prefect server?

Hmmm that’s very dependent on the server and network config so hard for me to say - I’d look up port forwarding for whatever service you’re using

Sorry, what's "service" mean in this context? It's running on a Kubernetes cluster in AWS, are either of those things relevant?

Yea, look at port forwarding to a pod in AWS kubernetes and you’ll just need to look up the dask dashboard default port number

I've seen this too. Be interested to hear what you find matta

Got an update here! Switched from threads to processes, and now it works fine. so, went from:

executor = DaskExecutor(
    cluster_kwargs={"processes": False}, 
    adapt_kwargs={"maximum": 6,},
)

to

executor = DaskExecutor(
    adapt_kwargs={"maximum": 6,},
)

I’m going to archive this thread for others to find — @Marvin archive “Mapped flow does not complete - switching to using processes resolves the issue”