Back again with what look like Prefect 2.0 API loa...
# ask-communityt
Tim Enders
10/26/2022, 6:02 PMBack again with what look like Prefect 2.0 API load issues? My first question is how would I debug the following? I don't have an exception being caught and I think this is a crash inside prefect code, not mine.
Copy code
13:00:10.647 | ERROR | Task run 'Get-Items-d8ed86f1-212' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.650 | ERROR | Task run 'Get-Items-d8ed86f1-15' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.652 | ERROR | Task run 'Get-Items-d8ed86f1-857' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.653 | ERROR | Task run 'Get-Items-d8ed86f1-774' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.654 | ERROR | Task run 'Get-Items-d8ed86f1-609' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.655 | ERROR | Task run 'Get-Items-d8ed86f1-444' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.657 | ERROR | Task run 'Get-Items-d8ed86f1-399' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.659 | ERROR | Task run 'Get-Items-d8ed86f1-700' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.660 | ERROR | Task run 'Get-Items-d8ed86f1-460' - Crash detected! Execution was cancelled by the runtime environment.Tim Enders
10/26/2022, 6:03 PMThis is attempting to run 1000 mapped tasks with the default ConcurrentRunner. I can have the flow run successfully with only 100 mapped tasks, but I have flows in Prefect 1.0 that are up to 45k mapped tasks.
z
Zanie
10/26/2022, 6:05 PM
Can you enable debug logs so we get a traceback for the crash?
gratitude thank you 1
Zanie
10/26/2022, 6:05 PM
What are you using for a backing server as well?
t
Tim Enders
10/26/2022, 6:05 PMI am connecting to my cloud API
Tim Enders
10/26/2022, 6:05 PMbut running locally
Tim Enders
10/26/2022, 6:06 PMHow would I enable DEBUG for the run... I am running the flow with
python path/to/my/file.pyz
Zanie
10/26/2022, 6:08 PM
PREFECT_DEBUG_MODE=1 or PREFECT_LOGGING_LEVEL=DEBUG
Zanie
10/26/2022, 6:08 PM
(environment variables, or
prefect config set SETTING=valuet
Tim Enders
10/26/2022, 6:58 PMGot a crash result out of there. I think this is what you are looking for. I will attach it as a text file
Tim Enders
10/26/2022, 6:59 PMCrash file. I have the whole stack trace, but it was emitting data so I don't really want to share it.
z
Zanie
10/26/2022, 7:16 PM
Thanks! What client version are you using?
Zanie
10/26/2022, 7:16 PM
We’ve made some recent changes to attempt to address this kind of connection error.
t
Tim Enders
10/26/2022, 7:18 PM2.6.4, I think that is up to date?
z
Zanie
10/26/2022, 8:04 PM
Yeah I’ll loop in some relevant parties. cc @Zach Angell / @Jake Kaplan
Zanie
10/26/2022, 8:04 PM
Just to check, you’re using Prefect Cloud without a proxy?
j
Jake Kaplan
10/26/2022, 8:07 PMTaking a look
🙏 1
t
Tim Enders
10/27/2022, 2:04 PMCorrect on the no proxy question
j
Jake Kaplan
10/27/2022, 9:07 PMSorry for the late follow up, I wasn't able to identify any errors in Cloud related to this. As best I can we never actually got the bulk of the requests and it looks like the errors seem to be socket based on the client side. Is there anything else you can tell us about your setup?
t
Tim Enders
10/27/2022, 9:09 PMIt is just my laptop with a Fiber connection (which is newish) hitting the cloud API. I wonder if my ISP is upset about the outgoing requests? 🤔
Tim Enders
10/27/2022, 9:09 PMI have upgraded to 2.6.5 today and things seem more stable? I haven't been able to test much today. I expect to do more tomorrow
Tim Enders
10/28/2022, 4:27 PM2.6.5 seems to be much more stable on the API. I am not running into any issues with my larger mapped tasks runs. Thank you all for the great work you are doing on this product!
z
Zanie
10/28/2022, 4:37 PM
You're welcome! We appreciate all the feedback.
15 Views