I'm using Prefect 2 to build a data pipeline and have a couple of general design questions:
⢠I assume we should be thinking about holding data in S3 and passing it between flows by reference ā i.e. rather than as parameters
⢠If so, should I use S3 blocks to store intermediate results, or interact with S3 directly? What are the trade-offs?
⢠I'm planning on using
Task.map to parallelise work (using dask); there's no equivalent for Flows, so I guess we should be thinking of parallelisation only happening
within flows, is that right?