Hello, just a general question. If in my data pipe...
# prefect-getting-started
k
Hello, just a general question. If in my data pipeline I use multiple flows on class methods, to update this object parameters, do they share between flows anyhow ? So for example I have my worker class and I call worker.update() in a flow (which changes object), the change won’t appear anywhere outside right? This wasn’t very straightforward and I think I fail to find any efficient methods to share the object between such flows. Does anyone have nice solution for that? I tried passing my object as a parameter to flow, but loading and unloading json of such a big object is extremely inefficient.
k
If you're operating on large data across multiple flows, it may make more sense to handle the data as an externally stored object that is read, modified, and written back between each step, or transformed in an entirely external system like a data warehouse. There's more advice on this subject in our big data guide: https://docs.prefect.io/latest/guides/big-data/
👍 1