Thread
#prefect-community
    Justin Liu

    Justin Liu

    1 year ago
    Hi all I followed this guide to set up my prefect enviornment with ecs fargate, and now I’m trying to use great expectations as part of the flow runs. As far as I know, the files for GE have to be on the same filesystem that’s running the flow, so does anyone know how I can get these files onto every ecs instance in a safe manner?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey sorry I haven’t had time to dig in to the GE stuff yet. You can add them to the image, but the container wouldn’t hold state if the files are modified. I think you need to see if you can use/persist it to S3 from the GE side.
    Justin Liu

    Justin Liu

    1 year ago
    Ahh ok, what do you mean by holding state? (sorry, new to this). we’ll try to look for a way to do that
    Kevin Kho

    Kevin Kho

    1 year ago
    Like if you run GE and the files change and get saved locally. You would lose them when the container spins down
    Zach Schumacher

    Zach Schumacher

    1 year ago
    you could mount an s3 bucket as a volume, potentially
    i know absolutely nothing about GE, tho
    s

    Spencer

    1 year ago
    For running stuff like this, I checkout the repo with the files to disk and run commands against the files as needed in the first task of my Flow. You'd want to create a deploy key on your repo and pass that as a secret to the checkout task.
    Justin Liu

    Justin Liu

    1 year ago
    this seems like a potential option thanks!