https://prefect.io logo
Title
m

Mathis Börner

07/24/2019, 10:04 AM
Another question I've got: I have to run the same flow for each file but different configs. I wanted to use prefect.context but struggled to save a flow with its context. So my idea was to have a Parameter('config') flowed by a manual_only task to "setup" the task: run the parameter task -> sleep the flow -> store the flow to fully process it later. It feels kind of off to do it this way, so is there a better way to achieve what I need?
c

Chris White

07/25/2019, 4:31 PM
Hi @Mathis Börner - sorry for the delay in responding to your question; I don’t think I fully understand your use case yet --> why do you need to pause for a long time right when you begin the flow? It seems like either: - you should have a Prefect task which does the required setup automatically - you should not run the flow until the setup is complete I’m confident I’m missing a nuance of your use case, but let me know!
m

Mathis Börner

07/29/2019, 6:08 AM
Hey, I want to make the tasks in my flow are reusable as possible, therefore I have tasks with rich configurable parameters. My idea was to provide those via prefect.context, but for me, it seemed like it is not possible to store a flow with its context. So I figured maybe I can run something like a "setup"-task which returns the config dict and stores the task. I wanted to produce a storable instance which contains the flow with its specific config. Probably there is a more straight-forward way in prefect or I simply messed up with storing the context.
c

Chris White

07/29/2019, 1:37 PM
Interesting; is there a reason using Prefect Parameters doesn’t work for you? Alternatively, you could simply attach the context to the flow like:
flow.my_context = context
and use
cloudpickle
to serialize the whole object