m

    Mathis Börner

    3 years ago
    Another question I've got: I have to run the same flow for each file but different configs. I wanted to use prefect.context but struggled to save a flow with its context. So my idea was to have a Parameter('config') flowed by a manual_only task to "setup" the task: run the parameter task -> sleep the flow -> store the flow to fully process it later. It feels kind of off to do it this way, so is there a better way to achieve what I need?
    Chris White

    Chris White

    3 years ago
    Hi @Mathis Börner - sorry for the delay in responding to your question; I don’t think I fully understand your use case yet --> why do you need to pause for a long time right when you begin the flow? It seems like either: - you should have a Prefect task which does the required setup automatically - you should not run the flow until the setup is complete I’m confident I’m missing a nuance of your use case, but let me know!
    m

    Mathis Börner

    3 years ago
    Hey, I want to make the tasks in my flow are reusable as possible, therefore I have tasks with rich configurable parameters. My idea was to provide those via prefect.context, but for me, it seemed like it is not possible to store a flow with its context. So I figured maybe I can run something like a "setup"-task which returns the config dict and stores the task. I wanted to produce a storable instance which contains the flow with its specific config. Probably there is a more straight-forward way in prefect or I simply messed up with storing the context.
    Chris White

    Chris White

    3 years ago
    Interesting; is there a reason using Prefect Parameters doesn’t work for you? Alternatively, you could simply attach the context to the flow like:
    flow.my_context = context
    and use
    cloudpickle
    to serialize the whole object