This is related to the questions asked by @Christopher Chong Tau Teng
I am trying to understand if Prefect Server can be used in our environment to manage multiple workflow streams.
We want to manage workflows in the following manner
1. Workflows are containerised
2. You can update the container image backing one workflow without affecting any other workflows
3. Some workflows depend on other workflows having completed correctly
4. Prefect itself is deployed in containers, not on bare-metal host
Call a container image which is to be used by one or more workflows a workflow engine.
My current understanding on how to do this is to separate out the flow scripts from the workflow engines, but I am a bit confused as to how I should actually implement this.
As a template example, consider a setup with
* Prefect Server running as as set of docker containers
* Two engine container images, tagged engine:1 and engine:2 respectively
* Two workflows WF1 and WF2 with asscociated scripts workflow_1.py and workflow_2.py
Then, the scenario plays out as follows:
1. both workflow_1.py and workflow_2.py are initially using engine:1.
2. an "issue" is identified with WF1 using engine:1. It has to be updated to use engine:2
3. workflow_2.py is updated to use engine:2 and re-registered with Prefect
4. WF1 now runs using engine:1 while WF2 runs using engine:2
How might this setup be configured in Prefect?