@Marvin I noticed that results are persisted only when flow finishes. But if it crashed they are not saved.

Sounds like you want to save your flow results even when it crashes. Let's figure out which version of Prefect you're using first. Which version are you using, Prefect 2.x or Prefect 3.x?

v3

Here's how to ensure your results are saved even if your flow crashes in Prefect 3.x: 1. Enable Result Persistence: You can enable result persistence on your flow or tasks by setting the

persist_result

attribute to

True

. 2. Global Persistence Setting: You can turn on result persistence globally by setting the

PREFECT_RESULTS_PERSIST_BY_DEFAULT

in your configuration.

bash
    prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT=true

3. Result Storage: Configure your result storage by using the

result_storage

keyword argument, which accepts an instantiated filesystem block or a block slug. You can find detailed instructions here. If you need further refinement or have additional concerns, let me know!

@Marvin it is enabled

There are some ongoing discussions and issues related to handling crashes and result persistence in Prefect. Here are some useful insights: 1. A GitHub issue mentions that cached tasks might crash when using certain libraries like

tenacity

for retries, which could indirectly relate to your problem if similar exceptions are affecting result persistence. 2. Flow-level caching and retry mechanisms are other areas being discussed. For deeper problem-solving: - Check the specific exceptions causing your flow to crash. - Review the GitHub issues related to crashes. If your configurations are all correct, this might be a bug. You could track or report it on the Prefect GitHub issues page. For further guidance on configurations, refer to the Prefect 3.x result persistence documentation.