Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Is there an option you can disk space set for fargate cluster? Along the lines of:
```"ephemeralStorage": {
   "sizeInGiB": 200
}```

Fargate cluster as resource manager right?

I guess it would be the volumes argument <https://cloudprovider.dask.org/en/latest/aws.html#elastic-container-service-ecs|here> ? But wouldn’t it be easier to just write into something like S3 instead? The workers can independently save/load data

I'm loading some files from S3 to disk for training ML models.

Not sure if it is volumes, you can create an ephemeral storage that only the container can see.

Volumes I guess would be something used for all the workers?

Yes they are (they use EFS), but I took a peek at the docs and don’t really have insight how to set it up. S3 would be simpler

I guess I could just load from s3 right? Skip a step. Thanks