06/13/2022, 6:47 PM
This message was deleted.
think of it this way: the orchestrator pattern is relevant when you want to trigger each of your flows via an API call. That's why deployment must be created in advance to manage dependencies more easily, there are plenty of ways to streamline the process: • automate via CI • build base images + more customized images when needed • virtual environments e.g. conda with SubprocessFlowRunner and if your tasks are only triggering Docker run commands, we will have a task for this in the future in (it doesn't exist yet) If you need more help than this explanation, I would be interested to hear more about your actual use case and the problem you're trying to solve without looking at implementation details, such as whether you actually need to run each task in a separate container - doing that is actually less beneficial than it may initially seem to be: it introduces a lot of overhead in the runtime, more storage and cost requirements, makes harder to pass data between tasks/processes - running everything in Docker is not a silver bullet so curious to hear more about your actual use case from a business perspective