Trying out pydantic models for re usable parameter groups an Prefect Community #prefect-ui

Trying out pydantic models for re-usable parameter...

Jacob Blanco

11/01/2024, 2:38 AM

Trying out pydantic models for re-usable parameter groups, and I'm seeing strange behaviour whereby the parameter's don't render at all in the UI. I either get the following in the Beta run page:

Add parameters to your flow definitions to make them reusable and configurable. Learn more

{}

in the current run page. The flow signature looks like this:

Copy code

@flow(name="Application Events")
def application_events(kafka_params: KafkaIngestionParams = KafkaIngestionParams()):

I can share the model definition but it contains a

bool

and an

int

with simple hardcoded values. I would expect to see something like this with the default values:

{ "param_a": 5, "param_b": true }

Nate

11/01/2024, 2:34 PM

hi @Jacob Blanco - can you show your model definition? and ideally also

prefect deployment inspect your-flow/your-deployment

Jacob Blanco

11/05/2024, 1:03 AM

KafkaIngestionParams model

Jacob Blanco

11/05/2024, 1:12 AM

Anonymized results of inspect.json

Jacob Blanco

11/05/2024, 2:08 AM

I just realized that I don't actually know how Prefect interprets defaults in Pydantic models. If you want the model to handle the defaults of individual parameters within should you call the flow as: a)

Copy code

def my_flow(params: AllOptionalsModel):

or b)

Copy code

def my_flow(params: OneRequiredModel = OneRequiredModel()):

Or something else? If I do a, I get an error stating that

params

is required and not provided even if all parameters there in are optional. If I do b I get an error because one of the fields are required when I construct

OneRequiredModel

so I'm not sure which is the correct approach. I think I may have the wrong mental framework for these Models. To my mind they act as a container of parameters, however it seems that Prefect thinks of them as just another kind of parameter. For me it's not meaningful to say that "the Model itself has default values" or that the "model is optional/required" in a way that's independent of the default values (if any) of it's constituent fields.

Jacob Blanco

11/12/2024, 1:50 AM

Any suggestions? I think that parameter groups are a great feature that deserves a ton more attention. It can be super helpful for standardization of flows which we are doing a ton in my organization. I can imagine it's also the case for others. Happy to help in any way I can but I'm not clever enough to understand how Prefect deals with these groups in detail on my own.

8 Views

Open in Slack

Previous Next