Hello prefect community, I wonder what is the best...
# prefect-community
l
Hello prefect community, I wonder what is the best way to share data within flow (prefect 1.x). What I’m trying to achieve is running a bunch of tasks using map (bit of a fire and forget approach - it just generates data someplace). These tasks run computations and report status. In certain situations it breaks and log would be produced. When there are many of these tasks, scrolling through the log is a bit cumbersome. My idea would be to use artifacts API and have a single page with all the errors summarized. I’d build that page incrementally by updating a single artifact with the info of what has failed, much like it is described here: https://docs-v1.prefect.io/orchestration/concepts/artifacts.html#creating-markdown-artifacts, but I’m not too sure how I could sync the content of the error list and the initial markdown_id across all of the mapped tasks and perhaps even with other tasks other than creating hell of passed parameters and extra tasks that would update the artifact page. My idea was something along the lines:
Copy code
tag = prefect.context.get("artifacts_tag_id")
        msgs: list = prefect.context.get("artifacts_content") or []
        msgs.append(str(err))
        content = create_err_artifact_from_list_of_errs(msgs)

        if tag:
            update_markdown_artifact(tag, content)
        else:
            tag = create_markdown_artifact(content)
            prefect.context["artifacts_tag_id"] = tag

        prefect.context["artifacts_content"] = msgs
but I found out that mutating the context from within a task is not recommended. So I wonder if there is any other prefect way of sharing these two ‘variables’ throughout the life of a prefect flow. Thank you very much!
m
Hey @Lukáš Pravda I think this might be helpful for what your looking at https://discourse.prefect.io/t/how-to-send-a-single-slack-notification-upon-failure-of-potentially-hundreds-of-dependent-upstream-tasks/324 It's technically related to slack notifications but it seems like you should be able to apply some of the concepts to what you're attempting.
👀 1