It seems like each flow could demand its own image in order to separate dependencies, which would mean creating an ECR repo for each workflow? I suppose this wouldn't be that difficult to script with Github Actions and aws-cli globbing a directory for workflow names?
k
Kevin Kho
04/14/2022, 8:16 PM
You can have multiple Flows in the same container, but a couple of links.
Check this, and more importantly, this. But also, I think the best practice is fixed name but versioned tag, though I understand if that gets tricky in CI/CD
m
Myles Steinhauser
04/14/2022, 8:16 PM
We’re using a single “mega-container” that contains all of our flows individual dependencies, which lives inside of a single ECR repo. The container has scripts for different entrypoint/commands for each Task.
upvote 1
j
Jason
04/14/2022, 8:18 PM
Yeah, it seems like any benefit of small containers with individual sets of dependencies won't be realized for a while given our small ETL setup right now. So Myles' setup probably makes sense for now.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.