https://prefect.io
Join Slack
Back again with what look like Prefect 2.0 API load issues? My first question is how would I debug t...
# ask-community
t
Back again with what look like Prefect 2.0 API load issues? My first question is how would I debug the following? I don't have an exception being caught and I think this is a crash inside prefect code, not mine.
Copy code
13:00:10.647 | ERROR   | Task run 'Get-Items-d8ed86f1-212' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.650 | ERROR   | Task run 'Get-Items-d8ed86f1-15' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.652 | ERROR   | Task run 'Get-Items-d8ed86f1-857' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.653 | ERROR   | Task run 'Get-Items-d8ed86f1-774' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.654 | ERROR   | Task run 'Get-Items-d8ed86f1-609' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.655 | ERROR   | Task run 'Get-Items-d8ed86f1-444' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.657 | ERROR   | Task run 'Get-Items-d8ed86f1-399' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.659 | ERROR   | Task run 'Get-Items-d8ed86f1-700' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.660 | ERROR   | Task run 'Get-Items-d8ed86f1-460' - Crash detected! Execution was cancelled by the runtime environment.
This is attempting to run 1000 mapped tasks with the default ConcurrentRunner. I can have the flow run successfully with only 100 mapped tasks, but I have flows in Prefect 1.0 that are up to 45k mapped tasks.
z
Can you enable debug logs so we get a traceback for the crash?
gratitude thank you 1
What are you using for a backing server as well?
t
I am connecting to my cloud API
but running locally
How would I enable DEBUG for the run... I am running the flow with 
python path/to/my/file.py
is it a flag, or an ENVVAR?
z
PREFECT_DEBUG_MODE=1 or PREFECT_LOGGING_LEVEL=DEBUG
(environment variables, or 
prefect config set SETTING=value
)
t
Got a crash result out of there. I think this is what you are looking for. I will attach it as a text file
Crash file. I have the whole stack trace, but it was emitting data so I don't really want to share it.
orion_crash_20221026.txt
z
Thanks! What client version are you using?
We’ve made some recent changes to attempt to address this kind of connection error.
t
2.6.4, I think that is up to date?
z
Yeah I’ll loop in some relevant parties. cc @Zach Angell / @Jake Kaplan
Just to check, you’re using Prefect Cloud without a proxy?
j
Taking a look
🙏 1
t
Correct on the no proxy question
j
Sorry for the late follow up, I wasn't able to identify any errors in Cloud related to this. As best I can we never actually got the bulk of the requests and it looks like the errors seem to be socket based on the client side. Is there anything else you can tell us about your setup?
t
It is just my laptop with a Fiber connection (which is newish) hitting the cloud API. I wonder if my ISP is upset about the outgoing requests? 🤔
I have upgraded to 2.6.5 today and things seem more stable? I haven't been able to test much today. I expect to do more tomorrow
2.6.5 seems to be much more stable on the API. I am not running into any issues with my larger mapped tasks runs. Thank you all for the great work you are doing on this product!
z
You're welcome! We appreciate all the feedback.
14 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by