Hi Prefect team, We had a question on monitoring flow progress when running with the DaskExecutor. We've made great progress running some initial data science workflows on Prefect Core -- last week we were able to map over a thousand items and run successfully on a 100-node Dask cluster. (What would have taken ~50 hours completed in half an hour -- so cool!) During a large run like that, it's hard to monitor progress since any logging in tasks gets output on Dask workers. We tried adding StateHandlers on our Tasks thinking we might be able to log from those and see the output in a Jupyter notebook where we're running the flow, but those also output on the workers. Is there a way to better monitor progress (e.g. what tasks have completed for mapped elements) of a flow while running with DaskExecutor?

Joe, awesome to hear the engine is working well for you! The challenge you’re describing — monitoring and introspecting lots of data-driven tasks in a distributed setting — is a major challenge and one of the primary reasons we formed a company to dedicate resources to solving it “right”. Prefect Cloud will go a long way toward solving this for you by providing an API, storage layer, and UI. Short of bringing in a “full” solution like that, we’ve seen folks have success with Dask’s own UI, and we’ve made some enhancements to the DaskExecutor that are aimed at making Prefect’s output align more closely with the Dask UI’s expectations.

Thanks @Jeremiah, much appreciated -- that all makes sense. We're still very interested in Prefect Cloud and now that we've made some progress on Core, trying out Cloud soon-ish would make a lot of sense. On integrations, we use Datadog today so definitely a +1 from us on being able to integrate with them for alerting, etc.

Fantastic, we’ve heard that request (DataDog) a couple times recently so we’re actively looking at it.