Danny Vilela
02/11/2021, 3:27 AMflow.parameters() -> Set[Parameter]
. But it seems like the implementation is “shallow,” in that it just checks for top-level tasks that are parameters (as opposed to recursively pulling parameters from tasks):
def parameters(self) -> Set[Parameter]:
"""
Returns any parameters of the flow.
Returns:
- set: a set of any Parameters in this flow
"""
return {p for p in self.tasks if isinstance(p, Parameter)}
Is there a reasonable way to extend the Flow class to search for all parameters declared within the Flow(..) as flow
context manager?Zanie
Flow.tasks
and that's how parameters end up there. Can you explain a bit more about the issue you're running into?Danny Vilela
02/11/2021, 10:18 PMPipelineConfig
dataclass). This/these config(s) can be shared across multiple tasks, and I do that to allow each task to uniformly access the same configuration settings as any other task. However, if I declare parameters within my flow and pass them to those config(s), they’re not bound to the flow’s tasks and thus not picked up from flow.parameters()
.
I’m fully open to the idea that this isn’t a best practice, and that it may even be an anti-pattern when working with prefect. I guess I’m wondering if this has come up before, and if it’s been solved. Or maybe I’m not properly applying the idea of parameterizing my flow?Zanie
config = create_config(param_a, ...)
where create_config(...) -> PipelineConfig
?Danny Vilela
02/11/2021, 10:27 PM