Hi all I followed this guide to set up my prefect enviornment with ecs fargate, and now I’m trying to use great expectations as part of the flow runs. As far as I know, the files for GE have to be on the same filesystem that’s running the flow, so does anyone know how I can get these files onto every ecs instance in a safe manner?

Hey sorry I haven’t had time to dig in to the GE stuff yet. You can add them to the image, but the container wouldn’t hold state if the files are modified. I think you need to see if you can use/persist it to S3 from the GE side.

Ahh ok, what do you mean by holding state? (sorry, new to this). we’ll try to look for a way to do that

Like if you run GE and the files change and get saved locally. You would lose them when the container spins down

you could mount an s3 bucket as a volume, potentially

For running stuff like this, I checkout the repo with the files to disk and run commands against the files as needed in the first task of my Flow. You'd want to create a deploy key on your repo and pass that as a secret to the checkout task.

this seems like a potential option thanks!