Darren Burton

    Darren Burton

    6 months ago
    I upgraded to My Orion instance last night to 2.0beta with the server running. It looked like everything worked from a UI perspective. However, when I run a flow they all fail with "Crash detected! Execution was interrupted by an unexpected exception. I am thinking I should probably stop all of the Prefect process and restart the Server. I don't see a command line or an option in the UI to stop or restart the Orion Server.
    Michael Adkins

    Michael Adkins

    6 months ago
    Where is your server runner?
    What error are your flows reporting when crashing?
    Darren Burton

    Darren Burton

    6 months ago
    The Server is running locally on a MacBook Pro
    Michael Adkins

    Michael Adkins

    6 months ago
    To restart your server, you can just Ctrl-C in the terminal that you ran
    prefect orion start
    from then run
    prefect orion start
    again
    Darren Burton

    Darren Burton

    6 months ago
    I am doing some more testing and will let you know. I killed all of the prefect processes including the API, Sever and Agent. I have restarted everything. That is exactly what I did. I am still getting "Crash detected! Execution was interrupted by an unexpected exception." I am not sure what the issue is just yet. I have a Flow that has six queries to a InfluxDB.
    I shut all Prefect processes down, and re-executed pip install --pre -U prefect to see if I can get my Python script to run from the command line. I am wondering if I need to reset the database.
    Michael Adkins

    Michael Adkins

    6 months ago
    I’ve just found out that including the
    --pre
    is including pre-release dependencies as well and httpx happens to have a beta out that we are not compatible with
    That may be the source of your errors
    Darren Burton

    Darren Burton

    6 months ago
    It still fails even running from the command line, each query fails with Crash detected! Execution was interrupted by an unexpected exception.
    Michael Adkins

    Michael Adkins

    6 months ago
    pip uninstall httpx
    and
    pip install httpx
    should get you back to a healthy state
    Darren Burton

    Darren Burton

    6 months ago
    Thanks let me try that
    Should I try re-installing with pip install --pre -U prefect after re-installing httpx?
    The script still failed to complete
    I have reset the database and stopped the server and the script still fails to complete.
    Michael Adkins

    Michael Adkins

    6 months ago
    Reinstalling with
    --pre
    will probably get you the pre-release httpx again which is the issue
    You’re going to have to share a crash traceback for me to help more though
    Darren Burton

    Darren Burton

    6 months ago
    Sure, how do I create that?
    Michael Adkins

    Michael Adkins

    6 months ago
    It should be in the logs for the failing flow run
    Or in the console
    Did you reinstall with
    --pre
    ?
    Darren Burton

    Darren Burton

    6 months ago
    I have now ran pip install -U prefect and running the query again and it still crashes.
    % python3 InfluxDB_Avg_Query-Prefect.py
    12:21:52.691 | INFO    | prefect.engine - Created flow run 'cocky-marmoset' for flow 'bin-plot-avg-main'
    12:21:52.692 | INFO    | Flow run 'cocky-marmoset' - Using task runner 'ConcurrentTaskRunner'
    12:21:52.863 | INFO    | Flow run 'cocky-marmoset' - Created subflow run 'black-manatee' for flow 'influx-db-queries'
    Executing InfuxDB queries to generate a dataframe for each application
    ======================================================================
    
    Starting CrowdSrike query...
    12:21:52.907 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_csavg-27d58d41-0' for task 'query_influxdb_csavg'
    completed, Starting nessues...
    12:21:52.951 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_nessusdavg-8341d2a1-0' for task 'query_influxdb_nessusdavg'
    completed, Starting Jamf...
    12:21:52.999 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_jamfavg-180dbbc1-0' for task 'query_influxdb_jamfavg'
    completed, Starting Java...
    12:21:53.045 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_javaavg-3c69037c-0' for task 'query_influxdb_javaavg'
    completed, Starting Tanium...
    12:21:53.093 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_taniumavg-e4411b29-0' for task 'query_influxdb_taniumavg'
    completed, Starting kerneld...
    12:21:53.142 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_kerneldavg-779bda23-0' for task 'query_influxdb_kerneldavg'
    Completed. Returning results to Flow
    java query time is 2022-03-16 12:21:58.730028
    tanium query time is 2022-03-16 12:23:25.526345
    CrowdStrike query time is 2022-03-16 12:23:30.659676
    jamf query time is 2022-03-16 12:23:31.996037
    nessusd query time is 2022-03-16 12:23:35.418833
    kerneld query time is 2022-03-16 12:23:38.463353
    12:23:38.524 | INFO    | Task run 'query_influxdb_csavg-27d58d41-0' - Crash detected! Execution was interrupted by an unexpected exception.
    12:23:38.546 | INFO    | Task run 'query_influxdb_nessusdavg-8341d2a1-0' - Crash detected! Execution was interrupted by an unexpected exception.
    12:23:38.568 | INFO    | Task run 'query_influxdb_jamfavg-180dbbc1-0' - Crash detected! Execution was interrupted by an unexpected exception.
    12:23:38.589 | INFO    | Task run 'query_influxdb_javaavg-3c69037c-0' - Crash detected! Execution was interrupted by an unexpected exception.
    12:23:38.609 | INFO    | Task run 'query_influxdb_taniumavg-e4411b29-0' - Crash detected! Execution was interrupted by an unexpected exception.
    12:23:38.632 | INFO    | Task run 'query_influxdb_kerneldavg-779bda23-0' - Crash detected! Execution was interrupted by an unexpected exception.
    /Users/darren.burton/opt/anaconda3/envs/Standard/lib/python3.7/site-packages/prefect/client.py:1190: UserWarning:
    
    No default storage has been set on the server. Using temporary local storage for results.
    
    12:23:38.699 | ERROR   | Flow run 'black-manatee' - Finished in state Failed('6/6 states failed.')
    Converting Query Results to bins and plotting in plotly - bin_plots function
    12:23:38.723 | INFO    | Flow run 'cocky-marmoset' - Created task run 'bin_plot_function-906e9ce7-0' for task 'bin_plot_function'
    Converting Combined Datframe Results info a Facet to plot with plotly - facet_plots function
    12:23:38.813 | INFO    | Flow run 'cocky-marmoset' - Created task run 'facet_plot_function-f75bc435-0' for task 'facet_plot_function'
    12:23:38.844 | ERROR   | Task run 'facet_plot_function-f75bc435-0' - Finished in state NotReady("Upstream task run 'None' did not reach a 'COMPLETED' state.", type=PENDING)
    12:23:38.844 | INFO    | Task run 'bin_plot_function-906e9ce7-0' - Crash detected! Execution was interrupted by an unexpected exception.
    /Users/darren.burton/opt/anaconda3/envs/Standard/lib/python3.7/site-packages/prefect/client.py:1190: UserWarning:
    
    No default storage has been set on the server. Using temporary local storage for results.
    
    12:23:38.914 | ERROR   | Flow run 'cocky-marmoset' - Finished in state Failed('2/3 states failed.')
    Michael Adkins

    Michael Adkins

    6 months ago
    Can you run this as
    PREFECT_DEBUG_MODE=1 python3 InfluxDB_Avg_Query-Prefect.py
    Darren Burton

    Darren Burton

    6 months ago
    Yes let me try that in a few min. I ended up uninstalling Prefect and using the specific version released yesterday
    pip install -U "prefect==2.0b1"
    That still failed, so I am running your command for additional debug info
    It doesn't like something. The Python code has not changed and has been running fine on the previous version.
    Michael Adkins

    Michael Adkins

    6 months ago
    Ah this is some silly behavior where they’ve overridden the equality operator
    You can use
    prefect.utilities.collections.quote
    to get around this for now
    i.e.
    return quote(my_object)
    from your task then where you consume it do
    my_object = result.unquote()
    This will be fixed in the next release, but shame on numexpr for breaking the equality operator
    Darren Burton

    Darren Burton

    6 months ago
    Thank you. I will try that. Do I need to import additional Prefect libraries to use it, if so do you have a reference?
    Michael Adkins

    Michael Adkins

    6 months ago
    You can just
    from prefect.utilities.collections import quote
    You can also just return your object wrapped in any other object, like
    {"value": my_dataframe}
    Darren Burton

    Darren Burton

    6 months ago
    Thanks @Michael Adkins I was able to make the appropriate modifications to my Python code and get a run to complete. BTW would you happen to have a link to how to modify the code to use Prefect Cloud 2.0 beta? I wasn't finding it earlier.
    please disregard, I just found it: https://orion-docs.prefect.io/ui/cloud/
    Michael Adkins

    Michael Adkins

    6 months ago
    Yep
    I’d recommend creating a separate profile for cloud
    Darren Burton

    Darren Burton

    6 months ago
    at the Python virtual environment level?
    Michael Adkins

    Michael Adkins

    6 months ago
    Ah no we have profile support e.g.
    prefect profile create cloud
    then
    prefect profile use cloud
    and
    prefect cloud login …
    You can easily switch back to local runs then
    Darren Burton

    Darren Burton

    6 months ago
    I thought I had solved my problem, but apparently I didn't. I am still trying to figure out the proper syntax to quote and unquote the dataframe to get the flow to work. Would have a more detailed example?
    Michael Adkins

    Michael Adkins

    6 months ago
    I'd just wrap it in a list or dict
    You just need to hide the dataframe return value inside another object
    Darren Burton

    Darren Burton

    6 months ago
    I am not familiar with that process so doing some searching
    Kevin Kho

    Kevin Kho

    6 months ago
    Could you show me the code you have Darren?
    Michael is saying do something like this:
    from prefect import flow, task 
    import pandas as pd
    from typing import Dict
    
    @task
    def get_data():
        df = pd.DataFrame({"a": [1,2,3], "b": [2,3,4]})
        return {"df": df}
    
    @task
    def add_col(df: Dict):
        df = df["df"]
        df['c'] = 1
        return {"df": df}
    
    @flow
    def myflow():
        df = get_data()
        add_col(df)
    
    myflow()
    Darren Burton

    Darren Burton

    6 months ago
    Thanks. Iet me try that. So it is at the task level where the modification is required, correct?
    Kevin Kho

    Kevin Kho

    6 months ago
    Yes
    Darren Burton

    Darren Burton

    6 months ago
    Still getting errors - unknown type object now
    18:12:10.714 | INFO    | prefect.engine - Created flow run 'boisterous-millipede' for flow 'myflow'
    18:12:10.715 | INFO    | Flow run 'boisterous-millipede' - Using task runner 'ConcurrentTaskRunner'
    18:12:10.767 | INFO    | Flow run 'boisterous-millipede' - Created task run 'query_influxdb_javamaxtest-6df64fa8-0' for task 'query_influxdb_javamaxtest'
    18:12:15.673 | INFO    | Task run 'query_influxdb_javamaxtest-6df64fa8-0' - Crash detected! Execution was interrupted by an unexpected exception.
    /Users/darren.burton/opt/anaconda3/envs/Standard/lib/python3.7/site-packages/prefect/client.py:1190: UserWarning:
    
    No default storage has been set on the server. Using temporary local storage for results.
    
    18:12:15.750 | ERROR   | Flow run 'boisterous-millipede' - Finished in state Failed('1/1 states failed.')
    Out[16]:
    Failed(message='1/1 states failed.', type=FAILED, result=[Crashed(message='Execution was interrupted by an unexpected exception.', type=FAILED, result=ValueError('unknown type object'), task_run_id=b0c42fa9-2724-4353-94e0-e0458f145b2b)], flow_run_id=23cad0de-f1a0-4f87-83cf-10c062407b90)
    @task
    def query_influxdb_javamaxtest():
        
    #Query: using Pandas DataFrame - Java Task
    
        javamax_df = query_api.query_data_frame('from(bucket: "bt-test")'
          '|> range(start: -31h, stop: now())'
          '|> filter(fn: (r) => r["_measurement"] == "bt_process_cpu") '                                     
          '|> filter(fn: (r) => r["_field"] == "Java_Max")'
          '|> drop(columns: ["_measurement"])'
          '|> yield(name: "last")' )
    
        """
        Close client
        """
        client.close()
        
        return {"df": javamax_df}
    
    
    @flow
    def myflow():
        df = query_influxdb_javamaxtest()
        
    
    myflow()
    Kevin Kho

    Kevin Kho

    6 months ago
    Ah ok we’ll need to look more into this behavior on our end. Sorry about that.
    Darren Burton

    Darren Burton

    6 months ago
    Thanks you so much and let me know if you come up with anything to try including upgrading to an pre-release version to test.
    Kevin Kho

    Kevin Kho

    6 months ago
    Did you try the quote like Michael suggested though?
    from lib2to3.pgen2.pgen import DFAState
    from prefect import flow, task 
    import pandas as pd
    from typing import Dict
    from prefect.utilities.collections import quote
    
    @task
    def get_data():
        df = pd.DataFrame({"a": [1,2,3], "b": [2,3,4]})
        return quote(df)
    
    @task
    def add_col(df):
        df = df.unquote()
        df['c'] = 1
        return quote(df)
    
    @flow
    def myflow():
        df = get_data()
        add_col(df)
    
    myflow()
    Darren Burton

    Darren Burton

    6 months ago
    I had tried, but I will try again as I have reduced my code to test with.
    It doesn't seem to work for me. I replaced the return statement and added lib2to3 import statement.
    Kevin Kho

    Kevin Kho

    6 months ago
    Ah ok yeah I think this is something we’ll have to work on next release. Michael may be able to respond with more info tomorrow
    Darren Burton

    Darren Burton

    6 months ago
    Thank you
    Michael Adkins

    Michael Adkins

    6 months ago
    We've already got a fix for the next release, not sure why quoting didn't work but I'd just roll back a version until next week
    The release is out now
    Darren Burton

    Darren Burton

    6 months ago
    Thanks Michael. I rolled back today, but look forward to doing the upgrade and testing either tomorrow or over the weekend.