Hello, we are evaluating prefect and have a particular use case in mind that requires us to
(1) Run high level workflow steps in "Project A"
(2) Run specific (smaller) steps of the above workflow in "Project B"
Is is possible to deploy an Agent to each project so that Prefect is managing the entire execution (we currently perform cross-project RPC calls but would like to move away from this approach)
03/28/2022, 9:48 AM
Prefect agents are lightweight processes that poll for new (scheduled) flow runs and they don't operate on projects but on labels.
• Projects in Prefect 1 provide a way of organizing your flows.
• Labels are a way to match specific flows with specific agents
It looks like labels are what you need in your use case.
If you are looking to run different workflow steps on different machines (agents), you could spin up a Prefect agent on each machine, build your workflow logic in a subflow (independent normal flow but one that gets called from another flow rather than running on schedule), and orchestrate it from a parent flow. Here is an example of how you can do it: https://discourse.prefect.io/t/can-i-run-a-flow-of-flows-that-triggers-each-child-flow-on-a-different-machine/180
03/28/2022, 11:10 AM
OK thank you Anna - ah I realised I used a potentially confusing term (Project - I didnt mean "Prefect Project", I meant a separate "Project Account" with its own security boundary). In any case the sub-graph solution sounds perfect!