Hello, just a general question. If in my data pipeline I use multiple flows on class methods, to update this object parameters, do they share between flows anyhow ?
So for example I have my worker class and I call worker.update() in a flow (which changes object), the change won’t appear anywhere outside right? This wasn’t very straightforward and I think I fail to find any efficient methods to share the object between such flows. Does anyone have nice solution for that? I tried passing my object as a parameter to flow, but loading and unloading json of such a big object is extremely inefficient.
k
Kevin Grismore
01/06/2024, 3:49 PM
If you're operating on large data across multiple flows, it may make more sense to handle the data as an externally stored object that is read, modified, and written back between each step, or transformed in an entirely external system like a data warehouse.
There's more advice on this subject in our big data guide: https://docs.prefect.io/latest/guides/big-data/
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.