Hello! I was wondering if anyone had any thoughts / boilerplate to dynamically generate tasks from a...
w
Hello! I was wondering if anyone had any thoughts / boilerplate to dynamically generate tasks from a dict or json file. The keys would be the task name and the parameters would be the values. This wouldn’t be a mapping use case because the order of execution has to be sequential and comes from the map iterable. Ideally once the tasks are generated (maybe with a loop and prefect futures and .submit in the flow?) it would display in the UI. The use case is onboarding a team that has develop a robust config based job runner for their ML lifecycle with hydra. The goal would be to add prefect on top so it can schedule but also add durability with persistence and caching?
f
Not 100% sure but with_options should help you
At least to set task name