I mainly use Prefect for forecasting. An important feature that I am looking for is to pass data (large pandas dataframe of about 1.5GB in csv, or numpy array) between tasks. Does Prefect already support serialising significant data between tasks? I saw something called PandasSerializer, but not sure what it is for.

Hey @Kha Nguyen, for temporary resources, check this link . You can use Prefect to spinup the hardware are use that more the jobs. This way, the memory management is pushed to Dask. Prefect does not handle garbage collection well at the moment. We do have users that write out to S3 and load it in for downstream tasks. Some users code it on their own with boto3, and some use our Results API , which has an S3Result which uses boto3 underneath. PandasSerializer is used in conjunction with the ResultsAPI to serialize before saving.

Hi Kevin, I did not plan to move away from ECS agent. I simply explore all the options. The Dask cluster example provided only create a Local Dask cluster. I simply thought that if I can provision a real distributed dask cluster (with many EC2 instances) on demand (as Fargate tasks), then I can still push the workload to dask cluster instead of using an ECS cluster directly.