https://prefect.io logo
Title
a

Amelie Froessl

09/08/2022, 9:58 AM
hello everyone 🙂 . I am trying to use the parameters to run a Flow from the UI. My config parameters input is a pydantic BaseModel. However I have a more complex data structure in that they types of my elements in the conifg parameters are also pydantic BaseModels (something similar to here ). Or like this:
class Parameters_1(BaseModel, extra=Extra.forbid):
    a: int
    b: int
    c: str


class Parameters_2(BaseModel, extra=Extra.forbid):
    d: int
    e: int
    f: str


class Parameters_3(BaseModel, extra=Extra.forbid):
    g: int
    h: int
    i: str


class TestConfig(BaseModel, extra=Extra.forbid):
    parameters_1: Parameters_1
    parameters_2: Parameters_2
    parameters_3: Parameters_3
On the UI it shows up like in the attached screenshot. The input would be a dictionary of the values in the respecitve BaseModel like :
{"a":1, "b":2, "c": "hello"}
. ... which is not the nicest I guess. I was wondering if there is a way to nicely have the parameters of a sub BaseModel show up/ editable in the UI? Or if somebody has a suggestion on how this could be solved in a nicer way? Thanks in advance!
1
a

Anna Geller

09/08/2022, 11:38 AM
Welcome once again, Amelie! Could you show how you use it in your flow? is it on an ad-hoc flow run that you have difficulties or on deployment? a minimal reproducible example that we could run for troubleshooting would be much appreciated
a

Amelie Froessl

09/08/2022, 12:20 PM
Thank you @Anna Geller! It is a deployment. Attached is a flow that mimics the basic functionality of the parameter structure I would like to use. Then I run the below commands to create a deployment:
$ prefect deployment build ./test_flow.py:main -n paramter-test -q parameter-test-queue

$ prefect deployment apply main-deployment.yaml
In the UI then I would like to have all of the separate fields in the parameters show up/ customizable.. but it seems like I can only customize the entire parameters one by one...
a

Anna Geller

09/08/2022, 12:58 PM
That's true because each of those parameters gets sent as a separate JSON dictionary in the request payload. Perhaps you could explore separating those out into separate parameters instead of nested Pydantic classes?
Also, it looks like you would benefit a lot from changing your design so that those tasks that take those parameters are Subflows, this way you get benefit of Pydantic validation since task run input is not validated against Pydantic
a

Amelie Froessl

09/08/2022, 1:03 PM
I see ok! Ah interesting so the inputs per flow , if a pydantic class, will be validated during execution?
so I've updated the code now to look like the attached code... I'm getting all of the sub-parameters now, but as one big block of parameters (see screenshot). Is there a way to get them separated in sub-sections showing which parameter (1,2 or 3) they belong to, when editing the values?
a

Anna Geller

09/08/2022, 1:40 PM
yup exactly
not sure what sub sections you mean - I was suggesting splitting it so that a single flow doesn't need to cover all those subsections, instead each subflow takes the model it needs
there are probably 10 different ways to approach it so I'd encourage you to try out various setups with subflows, passing values via Blocks e.g. JSON block etc, and if you hit against something impossible to design with the current functionality, open a GitHub issue with a feature request showing what's missing.
a

Amelie Froessl

09/08/2022, 1:45 PM
ok I will have a look at the blocks and see if I can find a useful configuration. Thanks for your help already! 🙂
🙌 1
a

Anna Geller

09/08/2022, 1:51 PM
a link for feature request proposal
🙌 1
a

Amelie Froessl

09/08/2022, 1:56 PM
do you have example code somewhere that shows how to use a block as a customizable input parameter?
a

Anna Geller

09/08/2022, 2:02 PM
sure:
# setup
from prefect.blocks.system import JSON

block = JSON(value=dict(question="ultimate", answer=42))
block.save("demo", overwrite=True)

# use e.g. in a flow/task
x = JSON.load("demo")
print(x.value)
print(type(x.value))
🙌 1
you can modify the block from the UI at any time
🙌 1
a

Amelie Froessl

09/09/2022, 2:38 PM
hey @Anna Geller, I've been trying with a couple of different ways to pass simpler arguments via the UI and I can't seem to get anything to work... I am always running into issues with the type of the input values. Could you help me out? All I'm trying to do is pass a Dictionary to the main flow as 1 argument. Attached is my code and the
parameters.json
file. running it locally works no problem... but when i copy paste the content of the
parameters.json
into the parameters input field in the UI to run with custom parameter. The flow is giving me the below error: I've tried a couple of different ways to pass a simple json now to the flow as a customizable input.. but I just can't seem to find a way that works.
Traceback (most recent call last):
  File "/home/amelie/.cache/pypoetry/virtualenvs/prefect-pipelines-Ixm-msGT-py3.8/lib/python3.8/site-packages/prefect/engine.py", line 284, in retrieve_flow_then_begin_flow_run
    parameters = flow.validate_parameters(flow_run.parameters)
  File "/home/amelie/.cache/pypoetry/virtualenvs/prefect-pipelines-Ixm-msGT-py3.8/lib/python3.8/site-packages/prefect/flows.py", line 269, in validate_parameters
    raise ParameterTypeError.from_validation_error(exc) from None
prefect.exceptions.ParameterTypeError: Flow run received invalid parameters:
 - parameters: value is not a valid dict
a

Anna Geller

09/09/2022, 7:08 PM
as I mentioned, there are multiple ways to approach it, but if you hit against something impossible to design with the current functionality, open a GitHub issue with a feature request showing what's missing or not working
if you are getting to a point that your input parameter is a large dictionary which you manage with a JSON file, it might be easier to instead store it e.g. in S3 and in your parameter only point to the S3 location
a

Amelie Froessl

09/21/2022, 7:59 AM
ok thank you ! I will have a look. Sorry for the late reply.
👍 1