We want to use AWS ECS to run flows using AWS ElasticFileSystem (EFS) to store the data created during data processing. We are able to mount the EFS from EC2 instance, but we are failing terribly when trying to do so when using runner

prefect.run_configs.ECSRun

. Our flow keeps starting and never completes - not even printing single line of text. Is there any working example of AWS ECS based prefect runner storing data on mounted AWS EFS volume?

Hey @Jan Vlčinský, I’ll look into this tomorrow

I will share the repository with "hello world" example which I have tried to extend to use EFS. I shall have it done within an hour or sooner.

Sounds good

Gotcha. Did you try the container definition under number 5 here ? There seems to be a lot of networking to get it right though. You also add it as a volume.

We used this tutorial, but not exactly step by step (we could be "too smart"). I will try the tutorial step by step tomorrow morning with another colleague and see, what we learn from it.

What was hard? The networking?

Yes, networking, using proper avaliability zone, understanding security groups as definition of what to get in/out but also as locations, from which is EFS accessed (the 2nd security group use surprised me). We used single availability zone EFS and it did not match the zone of ECS.