https://prefect.io logo
Title
t

Tim Enders

10/26/2022, 6:02 PM
Back again with what look like Prefect 2.0 API load issues? My first question is how would I debug the following? I don't have an exception being caught and I think this is a crash inside prefect code, not mine.
13:00:10.647 | ERROR   | Task run 'Get-Items-d8ed86f1-212' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.650 | ERROR   | Task run 'Get-Items-d8ed86f1-15' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.652 | ERROR   | Task run 'Get-Items-d8ed86f1-857' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.653 | ERROR   | Task run 'Get-Items-d8ed86f1-774' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.654 | ERROR   | Task run 'Get-Items-d8ed86f1-609' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.655 | ERROR   | Task run 'Get-Items-d8ed86f1-444' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.657 | ERROR   | Task run 'Get-Items-d8ed86f1-399' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.659 | ERROR   | Task run 'Get-Items-d8ed86f1-700' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.660 | ERROR   | Task run 'Get-Items-d8ed86f1-460' - Crash detected! Execution was cancelled by the runtime environment.
This is attempting to run 1000 mapped tasks with the default ConcurrentRunner. I can have the flow run successfully with only 100 mapped tasks, but I have flows in Prefect 1.0 that are up to 45k mapped tasks.
z

Zanie

10/26/2022, 6:05 PM
Can you enable debug logs so we get a traceback for the crash?
:gratitude-thank-you: 1
What are you using for a backing server as well?
t

Tim Enders

10/26/2022, 6:05 PM
I am connecting to my cloud API
but running locally
How would I enable DEBUG for the run... I am running the flow with
python path/to/my/file.py
is it a flag, or an ENVVAR?
z

Zanie

10/26/2022, 6:08 PM
PREFECT_DEBUG_MODE=1 or PREFECT_LOGGING_LEVEL=DEBUG
(environment variables, or
prefect config set SETTING=value
)
t

Tim Enders

10/26/2022, 6:58 PM
Got a crash result out of there. I think this is what you are looking for. I will attach it as a text file
Crash file. I have the whole stack trace, but it was emitting data so I don't really want to share it.
z

Zanie

10/26/2022, 7:16 PM
Thanks! What client version are you using?
We’ve made some recent changes to attempt to address this kind of connection error.
t

Tim Enders

10/26/2022, 7:18 PM
2.6.4, I think that is up to date?
z

Zanie

10/26/2022, 8:04 PM
Yeah I’ll loop in some relevant parties. cc @Zach Angell / @Jake Kaplan
Just to check, you’re using Prefect Cloud without a proxy?
j

Jake Kaplan

10/26/2022, 8:07 PM
Taking a look
:thank-you: 1
t

Tim Enders

10/27/2022, 2:04 PM
Correct on the no proxy question
j

Jake Kaplan

10/27/2022, 9:07 PM
Sorry for the late follow up, I wasn't able to identify any errors in Cloud related to this. As best I can we never actually got the bulk of the requests and it looks like the errors seem to be socket based on the client side. Is there anything else you can tell us about your setup?
t

Tim Enders

10/27/2022, 9:09 PM
It is just my laptop with a Fiber connection (which is newish) hitting the cloud API. I wonder if my ISP is upset about the outgoing requests? 🤔
I have upgraded to 2.6.5 today and things seem more stable? I haven't been able to test much today. I expect to do more tomorrow
2.6.5 seems to be much more stable on the API. I am not running into any issues with my larger mapped tasks runs. Thank you all for the great work you are doing on this product!
z

Zanie

10/28/2022, 4:37 PM
You're welcome! We appreciate all the feedback.