I think I may have tripped over a bug in the Deployments UI Prefect Community #prefect-ui

I think I may have tripped over a bug in the Deplo...

Jan Malek

09/27/2023, 11:38 AM

I think I may have tripped over a bug in the Deployments UI in 2.13.2 (self-hosted). Deployments... stopped showing up on the web UI altogether, I'm getting this empty Deployments screen instead:

Deceivious

09/27/2023, 12:00 PM

Are the deployments registered in the database? What does the network developer tab say about the request?

Jan Malek

09/27/2023, 12:00 PM

Yes, they are registered; again, if I use the API endpoint rather than the UI one it works

Deceivious

09/27/2023, 12:00 PM

what about the network tab?

Jan Malek

09/27/2023, 12:01 PM

all green/200 =_=

Jenny

09/27/2023, 1:57 PM

Hi @Jan Malek - thanks for flagging. Annoyingly I can't reproduce this so we'll need a bit more info to figure out what's happening with your deployments. Are you able to try answer these questions: 1. This is self hosted right? 2. Are all your other screens working ok? 3. You see nothing amiss in the developer tab? 4. How many deployments do you have? 5. Any idea how old your oldest deployment is? 6. Anything else that might be a bit different about one of your deployments? If it's not too much hassle, it might be useful to have the content of the

api/deployments/filter

response too.

Jan Malek

09/27/2023, 2:00 PM

1. Yep, one GKE 2. Yes, perfectly fine 3. Nope, nothing 4. 27 as of today 5. They're no older than half a year at most; majority are a month or two old 6. No; this broke after adding a new Deployment, but I've tried rolling back the changes and the issue persists

🤔 2

Jan Malek

09/27/2023, 2:10 PM

I cannot provide a filter endpoint response, as that seems to be a POST endpoint; the server sits behind an IAP proxy and I don't have an easy way of getting through that

Jan Malek

09/27/2023, 2:19 PM

Oh - one more datapoint. I've harvested the Deployment UUID from the logs and tried to manually construct the UI endpoint. It returns a blank page. The API endpoint for the same UUID is fine.

Jan Malek

09/27/2023, 3:29 PM

Further datapoint - I've now added a Schedule to the Deployment, and it is getting picked up and scheduled appropriately... but the Flow tab doesn't see any Deployments for itself either

Jenny

09/27/2023, 3:36 PM

Oh that’s odd. Are you able to send me an MRE of how you created that deployment?

Jan Malek

09/27/2023, 3:38 PM

Copy code

@sync_compatible
async def apply_deployment_config(
    flow_yaml: anyio.Path,
    upload: bool = False,
    result_dict_ref: typing.Optional[typing.Dict[str, Deployment]] = None,
):
    try:
        flow_yaml_path_str = str(flow_yaml)
        print_and_log(f"Loading {flow_yaml.name}...")
        yaml_deployment = await Deployment.load_from_yaml(flow_yaml_path_str)

        if not yaml_deployment:
            return None, yaml_deployment

        deployment_id = await yaml_deployment.apply(upload=upload)

    except Exception as Err:
        logging.exception(Err)
        return None, None

    else:
        result = deployment_id, yaml_deployment

        if result_dict_ref is not None:
            result_dict_ref[deployment_id] = yaml_deployment

        print_and_log(f"APPLIED {flow_yaml.name}: {deployment_id} ({yaml_deployment})")
        return result

Is the workhorse; it had been working for multiple Deployments for a long while

Jan Malek

09/27/2023, 4:29 PM

Another interesting detail; I'm not sure if this is actually related to the app or just our setup, but I'm getting blank pages with the following error on refreshing a page sometimes:

Jan Malek

09/27/2023, 4:29 PM

I've engaged our DevOps team to check if it's not something on the proxy side of things

Jenny

09/27/2023, 5:20 PM

I'm sorry I'm not certain I can work with that MRE. Do you have the payload you get back for that deployment when you query for it using the API? I'm guessing there's something in it that the UI is mishandling so if I can see what it's in the deployment fields (even if you sanitize it where necessary) we can maybe figure out what's going on.

Jan Malek

09/28/2023, 10:09 AM

@Jenny I think we've managed to figure out the root cause - there is a JavaScript error. By breakpointing into the logic, we could see the missing Deployments in the JS variables, but the bug causes the rendering to fail

Jan Malek

09/28/2023, 10:10 AM

Copy code

Error: getSchemaPropertyServiceConstructor missing case for float
    P6t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    lp <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    Xy <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    _6t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    J0 <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:64>
    J0 <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:64>
    _6t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    Xy <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    j6t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    e_t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    e_t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    c_t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    u_t <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    map <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    LQt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:67>
    map <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:93>
    getDeployments <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:64>
    response <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:59>
    run <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    execute <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:59>
    subscribe <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:59>
    subscribe <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:59>
    Oe <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:59>
    setup <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/Deployments-01547d48.js:1>
    vh <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Jtt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Ctt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    wt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    st <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    R <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    ce <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    run <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    update <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    afterLeave <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Yt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    _leaveCb <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    A <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    wt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    r <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    b <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    T <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    setTimeout handler*$T <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    onLeave <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    KT <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    KT <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    onLeave <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    vh <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    jc <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    U <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    k <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    leave <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    ie <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    zt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    At <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Mt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    At <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Mt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    At <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    R <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    ce <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    run <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    update <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Zt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    st <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    R <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    ce <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    run <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    update <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    Zt <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    st <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    R <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    ce <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    run <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
    update <https://ki-data-prefect.dev.nonprod.ki-in.com/assets/index-191a1ac6.js:1>
index-191a1ac6.js:59:14310

Error for reference

Jan Malek

09/28/2023, 10:23 AM

Okay, managed to find the issue!

Jan Malek

09/28/2023, 10:25 AM

One of the Deployments had a bad type (

float

) in the parameters schema. This causes a fatal exception in the rendering logic for Deployments. Deleting the YAML didn't solve the problem, because the bad type already made it to the backing DB.

Jan Malek

09/28/2023, 10:27 AM

Ideally this loop would have more granular error-handling that would prevent one bad Deployment's effects from spilling over to the whole page; I can make a GH issue for it

Jenny

09/28/2023, 5:45 PM

Ah thank you for tracking that down. Yes some better error handling would be good there. Also flagging that we now have opt-in server-side enforcement of of deployment parameter schemas which might be helpful in future?

2 Views

Open in Slack

Previous Next