Hi everyone, I'm trying to use Prefect for our ML...
# prefect-getting-started
m
Hi everyone, I'm trying to use Prefect for our ML pipelines. We train our ~20 (and growing) models on a weekly basis and I'm experimenting with some setups that could make sense. Our pipeline runs different models by providing a configuration file. Conceptually, I would say that it would make sense to have a single deployment that defines a flow that takes the configuration as an input and is then scheduled to be run for each configuration. However, providing parameters in the schedule is not supported. Alternatively, I wanted to create a flow that launches other flows and adds them to the work queue but I couldn't find how to do this in the documentation. Does anyone know how to do this or have an alternative approach. I suppose that we are not the first team that has a ML pipeline that runs for different models. Best, Mark
b
However, providing parameters in the schedule is not supported.
This is seems to a be a pretty popular request from our users. Would you mind adding a +1 to this feature request in GitHub?
Alternatively, I wanted to create a flow that launches other flows and adds them to the work queue but I couldn't find how to do this in the documentation.
This is totally possible. I think the orchestrator/worker pattern may help you out here. Here's a link to a github repo which you can reference as an example!
m
Thanks for the response. The worker pattern doesn't seem ideal because it seems that flows are more observable than tasks. Furthermore, this approach complicates launching ad hoc jobs. The per-schedule parameter feature seems ideal for this use-case because it makes use of flows, only requires a single deployment and supports ad hoc pipeline runs via the "quick run" button of a flow. For now, I will just create a deployment for each model even though it's not the most ergonomic solution.