I've started using prefect and I think its a great api for managing workflows (I see it as a human friendly wrapper around dask orchestration). Thanks for excellent documentation and a super useful tool!
I'm trying to determine how best to use prefect within an orchestration workflow design.
The workflow is essentially three generic steps:
1. spinning up various services (various web servers and processes)
2. manage these services: state, variables/properties, tasks (prefect seems to fit this, although I may need to find a good way to store state/data)
3. detect failure/errors in services and trigger redundancy/recovery tasks
I am torn on how to use prefect with 1 and 3. I don't think using prefect alone is the best solution, but I also think prefect(dask scheduler/executor) is a good solution for starting services (1) and handling triggers (3). However I'm not sure if prefect is ideal for active monitoring. Is there a prefect usecase for monitoring log files/services, or if there is a better tool for this job? I didn't see any mention of monitoring within the prefect core. I'm not talking about monitoring specific prefect tasks which is obvious that prefect should monitor its own tasks, more so on monitoring errors/issues in services.
If prefect isn't ideal for active monitoring of services, I envision using something like prometheus to watch data/logs from the services and trigger redundancy/recovery tasks in prefect if prometheus alerts or detects failures/issues. Are there any ideas for a better integration of prefect into this workflow?