https://prefect.io logo
Powered by
Title
d

Darren Burton

03/16/2022, 3:24 PM
I upgraded to My Orion instance last night to 2.0beta with the server running. It looked like everything worked from a UI perspective. However, when I run a flow they all fail with "Crash detected! Execution was interrupted by an unexpected exception. I am thinking I should probably stop all of the Prefect process and restart the Server. I don't see a command line or an option in the UI to stop or restart the Orion Server.
z

Zanie

03/16/2022, 3:28 PM
Where is your server runner?
What error are your flows reporting when crashing?
d

Darren Burton

03/16/2022, 3:49 PM
The Server is running locally on a MacBook Pro
z

Zanie

03/16/2022, 3:52 PM
To restart your server, you can just Ctrl-C in the terminal that you ran 
prefect orion start
from then run 
prefect orion start
again
d

Darren Burton

03/16/2022, 3:54 PM
I am doing some more testing and will let you know. I killed all of the prefect processes including the API, Sever and Agent. I have restarted everything. That is exactly what I did. I am still getting "Crash detected! Execution was interrupted by an unexpected exception." I am not sure what the issue is just yet. I have a Flow that has six queries to a InfluxDB.
I shut all Prefect processes down, and re-executed pip install --pre -U prefect to see if I can get my Python script to run from the command line. I am wondering if I need to reset the database.
z

Zanie

03/16/2022, 4:17 PM
I’ve just found out that including the 
--pre
is including pre-release dependencies as well and httpx happens to have a beta out that we are not compatible with
That may be the source of your errors
d

Darren Burton

03/16/2022, 4:17 PM
It still fails even running from the command line, each query fails with Crash detected! Execution was interrupted by an unexpected exception.
z

Zanie

03/16/2022, 4:18 PM
pip uninstall httpx
and 
pip install httpx
should get you back to a healthy state
d

Darren Burton

03/16/2022, 4:18 PM
Thanks let me try that
Should I try re-installing with pip install --pre -U prefect after re-installing httpx?
The script still failed to complete
I have reset the database and stopped the server and the script still fails to complete.
z

Zanie

03/16/2022, 4:52 PM
Reinstalling with 
--pre
will probably get you the pre-release httpx again which is the issue
You’re going to have to share a crash traceback for me to help more though
d

Darren Burton

03/16/2022, 5:03 PM
Sure, how do I create that?
z

Zanie

03/16/2022, 5:07 PM
It should be in the logs for the failing flow run
Or in the console
Did you reinstall with 
--pre
?
d

Darren Burton

03/16/2022, 5:28 PM
I have now ran pip install -U prefect and running the query again and it still crashes.
% python3 InfluxDB_Avg_Query-Prefect.py
12:21:52.691 | INFO    | prefect.engine - Created flow run 'cocky-marmoset' for flow 'bin-plot-avg-main'
12:21:52.692 | INFO    | Flow run 'cocky-marmoset' - Using task runner 'ConcurrentTaskRunner'
12:21:52.863 | INFO    | Flow run 'cocky-marmoset' - Created subflow run 'black-manatee' for flow 'influx-db-queries'
Executing InfuxDB queries to generate a dataframe for each application
======================================================================

Starting CrowdSrike query...
12:21:52.907 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_csavg-27d58d41-0' for task 'query_influxdb_csavg'
completed, Starting nessues...
12:21:52.951 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_nessusdavg-8341d2a1-0' for task 'query_influxdb_nessusdavg'
completed, Starting Jamf...
12:21:52.999 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_jamfavg-180dbbc1-0' for task 'query_influxdb_jamfavg'
completed, Starting Java...
12:21:53.045 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_javaavg-3c69037c-0' for task 'query_influxdb_javaavg'
completed, Starting Tanium...
12:21:53.093 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_taniumavg-e4411b29-0' for task 'query_influxdb_taniumavg'
completed, Starting kerneld...
12:21:53.142 | INFO    | Flow run 'black-manatee' - Created task run 'query_influxdb_kerneldavg-779bda23-0' for task 'query_influxdb_kerneldavg'
Completed. Returning results to Flow
java query time is 2022-03-16 12:21:58.730028
tanium query time is 2022-03-16 12:23:25.526345
CrowdStrike query time is 2022-03-16 12:23:30.659676
jamf query time is 2022-03-16 12:23:31.996037
nessusd query time is 2022-03-16 12:23:35.418833
kerneld query time is 2022-03-16 12:23:38.463353
12:23:38.524 | INFO    | Task run 'query_influxdb_csavg-27d58d41-0' - Crash detected! Execution was interrupted by an unexpected exception.
12:23:38.546 | INFO    | Task run 'query_influxdb_nessusdavg-8341d2a1-0' - Crash detected! Execution was interrupted by an unexpected exception.
12:23:38.568 | INFO    | Task run 'query_influxdb_jamfavg-180dbbc1-0' - Crash detected! Execution was interrupted by an unexpected exception.
12:23:38.589 | INFO    | Task run 'query_influxdb_javaavg-3c69037c-0' - Crash detected! Execution was interrupted by an unexpected exception.
12:23:38.609 | INFO    | Task run 'query_influxdb_taniumavg-e4411b29-0' - Crash detected! Execution was interrupted by an unexpected exception.
12:23:38.632 | INFO    | Task run 'query_influxdb_kerneldavg-779bda23-0' - Crash detected! Execution was interrupted by an unexpected exception.
/Users/darren.burton/opt/anaconda3/envs/Standard/lib/python3.7/site-packages/prefect/client.py:1190: UserWarning:

No default storage has been set on the server. Using temporary local storage for results.

12:23:38.699 | ERROR   | Flow run 'black-manatee' - Finished in state Failed('6/6 states failed.')
Converting Query Results to bins and plotting in plotly - bin_plots function
12:23:38.723 | INFO    | Flow run 'cocky-marmoset' - Created task run 'bin_plot_function-906e9ce7-0' for task 'bin_plot_function'
Converting Combined Datframe Results info a Facet to plot with plotly - facet_plots function
12:23:38.813 | INFO    | Flow run 'cocky-marmoset' - Created task run 'facet_plot_function-f75bc435-0' for task 'facet_plot_function'
12:23:38.844 | ERROR   | Task run 'facet_plot_function-f75bc435-0' - Finished in state NotReady("Upstream task run 'None' did not reach a 'COMPLETED' state.", type=PENDING)
12:23:38.844 | INFO    | Task run 'bin_plot_function-906e9ce7-0' - Crash detected! Execution was interrupted by an unexpected exception.
/Users/darren.burton/opt/anaconda3/envs/Standard/lib/python3.7/site-packages/prefect/client.py:1190: UserWarning:

No default storage has been set on the server. Using temporary local storage for results.

12:23:38.914 | ERROR   | Flow run 'cocky-marmoset' - Finished in state Failed('2/3 states failed.')
z

Zanie

03/16/2022, 5:44 PM
Can you run this as 
PREFECT_DEBUG_MODE=1 python3 InfluxDB_Avg_Query-Prefect.py
d

Darren Burton

03/16/2022, 5:44 PM
Yes let me try that in a few min. I ended up uninstalling Prefect and using the specific version released yesterday
pip install -U "prefect==2.0b1"
That still failed, so I am running your command for additional debug info
Prefect Run Debug Log.txt
It doesn't like something. The Python code has not changed and has been running fine on the previous version.
z

Zanie

03/16/2022, 6:29 PM
Ah this is some silly behavior where they’ve overridden the equality operator
You can use 
prefect.utilities.collections.quote
to get around this for now
i.e. 
return quote(my_object)
from your task then where you consume it do 
my_object = result.unquote()
This will be fixed in the next release, but shame on numexpr for breaking the equality operator
d

Darren Burton

03/16/2022, 6:40 PM
Thank you. I will try that. Do I need to import additional Prefect libraries to use it, if so do you have a reference?
z

Zanie

03/16/2022, 6:41 PM
You can just 
from prefect.utilities.collections import quote
You can also just return your object wrapped in any other object, like 
{"value": my_dataframe}
d

Darren Burton

03/16/2022, 7:47 PM
Thanks @Zanie I was able to make the appropriate modifications to my Python code and get a run to complete. BTW would you happen to have a link to how to modify the code to use Prefect Cloud 2.0 beta? I wasn't finding it earlier.
please disregard, I just found it: https://orion-docs.prefect.io/ui/cloud/
z

Zanie

03/16/2022, 7:50 PM
https://orion-docs.prefect.io/ui/cloud/ 🙂
Yep
I’d recommend creating a separate profile for cloud
d

Darren Burton

03/16/2022, 7:53 PM
at the Python virtual environment level?
z

Zanie

03/16/2022, 7:54 PM
Ah no we have profile support e.g. 
prefect profile create cloud
then 
prefect profile use cloud
and 
prefect cloud login …
https://orion-docs.prefect.io/concepts/settings/#configuration-profiles
You can easily switch back to local runs then
👍 1
d

Darren Burton

03/16/2022, 10:36 PM
I thought I had solved my problem, but apparently I didn't. I am still trying to figure out the proper syntax to quote and unquote the dataframe to get the flow to work. Would have a more detailed example?
z

Zanie

03/16/2022, 10:39 PM
I'd just wrap it in a list or dict
You just need to hide the dataframe return value inside another object
d

Darren Burton

03/16/2022, 10:42 PM
I am not familiar with that process so doing some searching
k

Kevin Kho

03/16/2022, 10:48 PM
Could you show me the code you have Darren?
Michael is saying do something like this: 
from prefect import flow, task 
import pandas as pd
from typing import Dict

@task
def get_data():
    df = pd.DataFrame({"a": [1,2,3], "b": [2,3,4]})
    return {"df": df}

@task
def add_col(df: Dict):
    df = df["df"]
    df['c'] = 1
    return {"df": df}

@flow
def myflow():
    df = get_data()
    add_col(df)

myflow()
d

Darren Burton

03/16/2022, 10:53 PM
Thanks. Iet me try that. So it is at the task level where the modification is required, correct?
k

Kevin Kho

03/16/2022, 10:54 PM
Yes
d

Darren Burton

03/16/2022, 11:22 PM
Still getting errors - unknown type object now
18:12:10.714 | INFO    | prefect.engine - Created flow run 'boisterous-millipede' for flow 'myflow'
18:12:10.715 | INFO    | Flow run 'boisterous-millipede' - Using task runner 'ConcurrentTaskRunner'
18:12:10.767 | INFO    | Flow run 'boisterous-millipede' - Created task run 'query_influxdb_javamaxtest-6df64fa8-0' for task 'query_influxdb_javamaxtest'
18:12:15.673 | INFO    | Task run 'query_influxdb_javamaxtest-6df64fa8-0' - Crash detected! Execution was interrupted by an unexpected exception.
/Users/darren.burton/opt/anaconda3/envs/Standard/lib/python3.7/site-packages/prefect/client.py:1190: UserWarning:

No default storage has been set on the server. Using temporary local storage for results.

18:12:15.750 | ERROR   | Flow run 'boisterous-millipede' - Finished in state Failed('1/1 states failed.')
Out[16]: 
Failed(message='1/1 states failed.', type=FAILED, result=[Crashed(message='Execution was interrupted by an unexpected exception.', type=FAILED, result=ValueError('unknown type object'), task_run_id=b0c42fa9-2724-4353-94e0-e0458f145b2b)], flow_run_id=23cad0de-f1a0-4f87-83cf-10c062407b90)
@task
def query_influxdb_javamaxtest():
    
#Query: using Pandas DataFrame - Java Task

    javamax_df = query_api.query_data_frame('from(bucket: "bt-test")'
      '|> range(start: -31h, stop: now())'
      '|> filter(fn: (r) => r["_measurement"] == "bt_process_cpu") '                                     
      '|> filter(fn: (r) => r["_field"] == "Java_Max")'
      '|> drop(columns: ["_measurement"])'
      '|> yield(name: "last")' )

    """
    Close client
    """
    client.close()
    
    return {"df": javamax_df}


@flow
def myflow():
    df = query_influxdb_javamaxtest()
    

myflow()
k

Kevin Kho

03/16/2022, 11:30 PM
Ah ok we’ll need to look more into this behavior on our end. Sorry about that.
d

Darren Burton

03/16/2022, 11:51 PM
Thanks you so much and let me know if you come up with anything to try including upgrading to an pre-release version to test.
k

Kevin Kho

03/16/2022, 11:53 PM
Did you try the quote like Michael suggested though? 
from lib2to3.pgen2.pgen import DFAState
from prefect import flow, task 
import pandas as pd
from typing import Dict
from prefect.utilities.collections import quote

@task
def get_data():
    df = pd.DataFrame({"a": [1,2,3], "b": [2,3,4]})
    return quote(df)

@task
def add_col(df):
    df = df.unquote()
    df['c'] = 1
    return quote(df)

@flow
def myflow():
    df = get_data()
    add_col(df)

myflow()
d

Darren Burton

03/16/2022, 11:55 PM
I had tried, but I will try again as I have reduced my code to test with.
It doesn't seem to work for me. I replaced the return statement and added lib2to3 import statement.
k

Kevin Kho

03/17/2022, 12:09 AM
Ah ok yeah I think this is something we’ll have to work on next release. Michael may be able to respond with more info tomorrow
d

Darren Burton

03/17/2022, 12:26 AM
Thank you
z

Zanie

03/17/2022, 12:36 AM
We've already got a fix for the next release, not sure why quoting didn't work but I'd just roll back a version until next week
The release is out now
d

Darren Burton

03/18/2022, 1:25 AM
Thanks Michael. I rolled back today, but look forward to doing the upgrade and testing either tomorrow or over the weekend.
#prefect-serverJoin Slack
Powered by