I m trying to pass a list as a parameter in the Prefect 2 3 Prefect Community #prefect-ui

I'm trying to pass a list as a parameter in the Pr...

Richard Alexander

09/29/2022, 1:58 PM

I'm trying to pass a list as a parameter in the Prefect 2.3 UI, and I can't seem to get the format right. I want to start the run via UI interface. I've used square and curly brackets... single and double quotes, but I continue to get the same error:

Copy code

ParameterTypeError('Flow run received invalid parameters:\n - part_list: value is not a valid list')

When starting a flow run via UI, what is the correct format that the UI expects when passing a list as a parameter?

✅ 1

Jenny

09/29/2022, 2:46 PM

Hi @Richard Alexander - are you able to upgrade to the latest version of Prefect? We made a lot of improvements to how the ui handles inputs in our recent releases.

Richard Alexander

09/29/2022, 2:59 PM

I can give that a shot @Jenny. Once I upgrade, what format should I use?

Richard Alexander

09/29/2022, 3:15 PM

I upgraded to 2.4.2 @Jenny, but I'm still having the same problem... I can't get the UI to accept a list. Not sure if it matters, but I'm specifically passing a list of strings. Any other ideas?

Jenny

09/29/2022, 3:21 PM

Are you using type hinting? You can see more in the docs - worth also checking the link in there to pydantic. https://docs.prefect.io/concepts/flows/

Zanie

09/29/2022, 3:21 PM

Can you share the output of

prefect flow-run inspect <your-flow-run>

Richard Alexander

09/29/2022, 3:26 PM

Yes, I have specified

part_list: string

Richard Alexander

09/29/2022, 3:26 PM

Here's the output of my last attempt @Zanie with curly brackets and double quotes, though I have tried square brackets too:

Copy code

FlowRun(
    id='91e20552-b28b-4f40-a7f3-10c49ff3d6be',
    created=DateTime(2022, 9, 29, 15, 13, 47, 577427, tzinfo=Timezone('+00:00')),
    updated=DateTime(2022, 9, 29, 15, 13, 51, 979492, tzinfo=Timezone('+00:00')),
    name='omicron4-minhiriath',
    flow_id='254e5f71-eeb0-45c0-a3e1-6239cea4946f',
    state_id='c2ae0c28-c3b7-4c25-a928-e685e1f6baa6',
    deployment_id='9f138df5-710c-4940-a52d-08ee52349cf2',
    work_queue_name='general',
    flow_version='d290e9f5a2cfbe5193231cb5645881c5',
    parameters={
        'part_list': '{"170-215", "66205-85", "3059258-01"}'
    },
    empirical_policy=FlowRunPolicy(retries=0, retry_delay=0),
    state_type=StateType.FAILED,
    state_name='Failed',
    expected_start_time=DateTime(2022, 9, 29, 15, 13, 47, 577235, tzinfo=Timezone('+00:00')),
    infrastructure_document_id='fa337e2c-92f3-4c42-b48e-2aed81a67be5',
    state=State(
        id='c2ae0c28-c3b7-4c25-a928-e685e1f6baa6',
        type=StateType.FAILED,
        name='Failed',
        timestamp=DateTime(2022, 9, 29, 15, 13, 51, 976706, tzinfo=Timezone('+00:00')),
        message="Validation of flow parameters failed with error: ParameterTypeError('Flow run received invalid parameters:\\n - part_list: value is not a valid list')",
        data=DataDocument(
            encoding='cloudpickle',
            blob=b'gAWVgQAAAAAAAACMEnByZWZlY3QuZXhjZXB0aW9uc5SMElBhcmFtZXRlclR5cGVFcnJvcpSTlIxN\nRmxvdyBydW4gcmVjZWl2ZWQgaW52YWxpZCBwYXJhbWV0ZXJzOgogLSBwYXJ0X2xpc3Q6IHZhbHVl\nIGlzIG5vdCBhIHZhbGlkIGxpc3SUhZRSlC4=\n'
        ),
        state_details=StateDetails(flow_run_id='91e20552-b28b-4f40-a7f3-10c49ff3d6be')
    )
)

Zanie

09/29/2022, 3:28 PM

Ah thanks! You should have the type be

List[str]

in your flow definition then in the UI you’ll want to pass it as

["foo", "bar"]

Zanie

09/29/2022, 3:28 PM

If you can share what you enter in the UI then the result of the inspection we should be able to get to the bottom of this 🙂

Richard Alexander

09/29/2022, 3:29 PM

Great, I'll give that a go first. Thanks for the help so far! ;-)

Richard Alexander

09/29/2022, 3:42 PM

Now it's treating the entire thing as a string @Zanie:

Copy code

flow_version='3eab00f5dd40a198fd2e4f61c712950b',
    parameters={
        'part_list': [
            '"170-215", "66205-85", "3059258-01"'
        ]
    },

I changed my method as you suggested:

Copy code

@flow()
def my_flow(part_list: list[str]):
	...

Richard Alexander

09/29/2022, 3:42 PM

And entered this in the UI:

Copy code

"170-215", "66205-85", "3059258-01"

Richard Alexander

09/29/2022, 3:46 PM

Whoops... I had a lowercase "l" in "list". That might be the problem...

Zanie

09/29/2022, 3:50 PM

Did you re-register the deployment after updating the flow?

Zanie

09/29/2022, 3:50 PM

It may need to be re-registered for the UI to know what the type should be

Richard Alexander

09/29/2022, 3:52 PM

I did, but I got a strange error:

Copy code

UserWarning: Block document has schema checksum sha256:d28332d78ac90882bfeeef79f98cd26c99cda6a7c6d3a48d56a6bd67d7360648 which does not match the schema checksum for class 'Process'. This indicates the schema has changed and this block may not load.
  self.infrastructure = Block._from_block_document(

Richard Alexander

09/29/2022, 3:52 PM

Not sure if that has a bearing on the issue.

Richard Alexander

09/29/2022, 3:54 PM

@Zanie I have tried a couple more times after fixing the "List" type hint, but the input is being treated as a single long string everytime:

Copy code

FlowRun(
    id='ada166b4-5279-449a-adc1-b0922c2d6b65',
    created=DateTime(2022, 9, 29, 15, 50, 0, 80785, tzinfo=Timezone('+00:00')),
    updated=DateTime(2022, 9, 29, 15, 50, 7, 965278, tzinfo=Timezone('+00:00')),
    name='theta5-magus-manifold',
    flow_id='254e5f71-eeb0-45c0-a3e1-6239cea4946f',
    state_id='386e6aa3-53b1-414b-a314-036061a56d32',
    deployment_id='9f138df5-710c-4940-a52d-08ee52349cf2',
    work_queue_name='general',
    flow_version='2db1960d59edb08a61304c30ce80861f',
    parameters={'part_list': ['["170-215", "66205-85", "3059258-01"]']},
    empirical_policy=FlowRunPolicy(retries=0, retry_delay=0),
    state_type=StateType.COMPLETED,
    state_name='Completed',
    run_count=1,
    expected_start_time=DateTime(2022, 9, 29, 15, 50, 0, 80471, tzinfo=Timezone('+00:00')),
    start_time=DateTime(2022, 9, 29, 15, 50, 6, 424794, tzinfo=Timezone('+00:00')),
    end_time=DateTime(2022, 9, 29, 15, 50, 7, 963240, tzinfo=Timezone('+00:00')),
    total_run_time=datetime.timedelta(seconds=1, microseconds=538446),
    estimated_run_time=datetime.timedelta(seconds=1, microseconds=538446),
    estimated_start_time_delta=datetime.timedelta(seconds=6, microseconds=344323),
    infrastructure_document_id='fa337e2c-92f3-4c42-b48e-2aed81a67be5',
    state=State(
        id='386e6aa3-53b1-414b-a314-036061a56d32',
        type=StateType.COMPLETED,
        name='Completed',
        timestamp=DateTime(2022, 9, 29, 15, 50, 7, 963240, tzinfo=Timezone('+00:00')),
        message='All states completed.',
        data=DataDocument(encoding='result', blob=b'{"key": "5eaebc2484eb4f2bb910f1091d98f5a5", "filesystem_document_id": "7fc03c9d-acbb-4246-9ef4-079a74f9d80f"}'),
        state_details=StateDetails(flow_run_id='ada166b4-5279-449a-adc1-b0922c2d6b65')
    )
)

Zanie

09/29/2022, 4:07 PM

Weird. Can you share a screenshot of how you’re entering it in the UI?

Richard Alexander

09/29/2022, 4:39 PM

Zanie

09/29/2022, 4:40 PM

Oh it looks like it knows its a list and it wants you to just input each value separately, cc @Jenny

Richard Alexander

09/29/2022, 4:40 PM

@Then when I press enter, I just noticed that it's already indicating that it is a single string:

Zanie

09/29/2022, 4:41 PM

You should be able to do just 1 -> Enter 2 -> Enter 3 -> Enter

Richard Alexander

09/29/2022, 4:42 PM

Oh man... I have a list size of 1K... I'd rather not do that! 😉

Richard Alexander

09/29/2022, 4:42 PM

But I will test that

Zanie

09/29/2022, 4:42 PM

Yeah sounds like we ought to provide a way to enter data in a raw format 😄

Richard Alexander

09/29/2022, 4:47 PM

Yes, looks like you're right. The only way to make it work currently is to enter the list one at a time. I would VERY much appreciate a raw format method. Nearly all of my flow runs have list parameters in the 1-10K range.

Richard Alexander

09/29/2022, 4:48 PM

Thanks for the help today Michael and Jenny! 😄

Zanie

09/29/2022, 4:48 PM

You can create runs outside of the UI with requests directly to the API

Zanie

09/29/2022, 4:48 PM

If that’d help in the meantime 🙂

Richard Alexander

09/29/2022, 4:48 PM

Oh... via command line? Or some other method?

Jenny

09/29/2022, 4:49 PM

If you can, I believe that if you don't specify it as a list of strings it should give you a json input.

Richard Alexander

09/29/2022, 4:50 PM

Right. Isn't that what I was dealing with in the beginning though? Before adding the type hint? I'll revert that and double check.

Zanie

09/29/2022, 4:51 PM

If it’s not specified as a list of strings the client won’t parse it from JSON into a list of strings at runtime

Zanie

09/29/2022, 4:52 PM

We’ll need the ability to do raw inputs in the UI to support this kind of use-case

Richard Alexander

09/29/2022, 4:52 PM

Right, I think that was my initial problem.

Zanie

09/29/2022, 4:54 PM

Yeah we rely on those type annotations to cast the parameters passed by the API to a run into Python datatypes

Zanie

09/29/2022, 4:54 PM

Copy code

import asyncio
from prefect import get_client

async def create_run():
    async with get_client() as client:
        flow_run = await client.create_flow_run_from_deployment(
            "<deployment-id>", parameters={"your-param": ["your", "big", "list"]}
        )
        print(flow_run.id)
        

asyncio.run(create_run)

Zanie

09/29/2022, 4:55 PM

We’re still working on parameter specification for the

prefect deployment run

command. Parsing JSON from the CLI is kind of a pain.

Jenny

09/29/2022, 6:33 PM

https://github.com/PrefectHQ/prefect/issues/7021 Feel free to add any extra info/context.

🙏 1

12 Views

Open in Slack

Previous Next