Hello! I was wondering if anyone had any thoughts / boilerplate to dynamically generate tasks from a dict or json file. The keys would be the task name and the parameters would be the values. This wouldn’t be a mapping use case because the order of execution has to be sequential and comes from the map iterable. Ideally once the tasks are generated (maybe with a loop and prefect futures and .submit in the flow?) it would display in the UI.
The use case is onboarding a team that has develop a robust config based job runner for their ML lifecycle with hydra. The goal would be to add prefect on top so it can schedule but also add durability with persistence and caching?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.