hello everyone :slightly_smiling_face: . I am tryi...
# prefect-ui
hello everyone 🙂 . I am trying to use the parameters to run a Flow from the UI. My config parameters input is a pydantic BaseModel. However I have a more complex data structure in that they types of my elements in the conifg parameters are also pydantic BaseModels (something similar to here ). Or like this:
Copy code
class Parameters_1(BaseModel, extra=Extra.forbid):
    a: int
    b: int
    c: str

class Parameters_2(BaseModel, extra=Extra.forbid):
    d: int
    e: int
    f: str

class Parameters_3(BaseModel, extra=Extra.forbid):
    g: int
    h: int
    i: str

class TestConfig(BaseModel, extra=Extra.forbid):
    parameters_1: Parameters_1
    parameters_2: Parameters_2
    parameters_3: Parameters_3
On the UI it shows up like in the attached screenshot. The input would be a dictionary of the values in the respecitve BaseModel like :
{"a":1, "b":2, "c": "hello"}
. ... which is not the nicest I guess. I was wondering if there is a way to nicely have the parameters of a sub BaseModel show up/ editable in the UI? Or if somebody has a suggestion on how this could be solved in a nicer way? Thanks in advance!
Welcome once again, Amelie! Could you show how you use it in your flow? is it on an ad-hoc flow run that you have difficulties or on deployment? a minimal reproducible example that we could run for troubleshooting would be much appreciated
Thank you @Anna Geller! It is a deployment. Attached is a flow that mimics the basic functionality of the parameter structure I would like to use. Then I run the below commands to create a deployment:
Copy code
$ prefect deployment build ./test_flow.py:main -n paramter-test -q parameter-test-queue

$ prefect deployment apply main-deployment.yaml
In the UI then I would like to have all of the separate fields in the parameters show up/ customizable.. but it seems like I can only customize the entire parameters one by one...
That's true because each of those parameters gets sent as a separate JSON dictionary in the request payload. Perhaps you could explore separating those out into separate parameters instead of nested Pydantic classes?
Also, it looks like you would benefit a lot from changing your design so that those tasks that take those parameters are Subflows, this way you get benefit of Pydantic validation since task run input is not validated against Pydantic
I see ok! Ah interesting so the inputs per flow , if a pydantic class, will be validated during execution?
so I've updated the code now to look like the attached code... I'm getting all of the sub-parameters now, but as one big block of parameters (see screenshot). Is there a way to get them separated in sub-sections showing which parameter (1,2 or 3) they belong to, when editing the values?
yup exactly
not sure what sub sections you mean - I was suggesting splitting it so that a single flow doesn't need to cover all those subsections, instead each subflow takes the model it needs
there are probably 10 different ways to approach it so I'd encourage you to try out various setups with subflows, passing values via Blocks e.g. JSON block etc, and if you hit against something impossible to design with the current functionality, open a GitHub issue with a feature request showing what's missing.
ok I will have a look at the blocks and see if I can find a useful configuration. Thanks for your help already! 🙂
🙌 1
a link for feature request proposal
🙌 1
do you have example code somewhere that shows how to use a block as a customizable input parameter?
Copy code
# setup
from prefect.blocks.system import JSON

block = JSON(value=dict(question="ultimate", answer=42))
block.save("demo", overwrite=True)

# use e.g. in a flow/task
x = JSON.load("demo")
🙌 1
you can modify the block from the UI at any time
🙌 1
hey @Anna Geller, I've been trying with a couple of different ways to pass simpler arguments via the UI and I can't seem to get anything to work... I am always running into issues with the type of the input values. Could you help me out? All I'm trying to do is pass a Dictionary to the main flow as 1 argument. Attached is my code and the
file. running it locally works no problem... but when i copy paste the content of the
into the parameters input field in the UI to run with custom parameter. The flow is giving me the below error: I've tried a couple of different ways to pass a simple json now to the flow as a customizable input.. but I just can't seem to find a way that works.
Copy code
Traceback (most recent call last):
  File "/home/amelie/.cache/pypoetry/virtualenvs/prefect-pipelines-Ixm-msGT-py3.8/lib/python3.8/site-packages/prefect/engine.py", line 284, in retrieve_flow_then_begin_flow_run
    parameters = flow.validate_parameters(flow_run.parameters)
  File "/home/amelie/.cache/pypoetry/virtualenvs/prefect-pipelines-Ixm-msGT-py3.8/lib/python3.8/site-packages/prefect/flows.py", line 269, in validate_parameters
    raise ParameterTypeError.from_validation_error(exc) from None
prefect.exceptions.ParameterTypeError: Flow run received invalid parameters:
 - parameters: value is not a valid dict
as I mentioned, there are multiple ways to approach it, but if you hit against something impossible to design with the current functionality, open a GitHub issue with a feature request showing what's missing or not working
if you are getting to a point that your input parameter is a large dictionary which you manage with a JSON file, it might be easier to instead store it e.g. in S3 and in your parameter only point to the S3 location
ok thank you ! I will have a look. Sorry for the late reply.
👍 1