Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
t
Tim Enders
10/26/2022, 6:02 PMBack again with what look like Prefect 2.0 API load issues? My first question is how would I debug the following? I don't have an exception being caught and I think this is a crash inside prefect code, not mine.
13:00:10.647 | ERROR | Task run 'Get-Items-d8ed86f1-212' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.650 | ERROR | Task run 'Get-Items-d8ed86f1-15' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.652 | ERROR | Task run 'Get-Items-d8ed86f1-857' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.653 | ERROR | Task run 'Get-Items-d8ed86f1-774' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.654 | ERROR | Task run 'Get-Items-d8ed86f1-609' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.655 | ERROR | Task run 'Get-Items-d8ed86f1-444' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.657 | ERROR | Task run 'Get-Items-d8ed86f1-399' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.659 | ERROR | Task run 'Get-Items-d8ed86f1-700' - Crash detected! Execution was cancelled by the runtime environment.
13:00:10.660 | ERROR | Task run 'Get-Items-d8ed86f1-460' - Crash detected! Execution was cancelled by the runtime environment.This is attempting to run 1000 mapped tasks with the default ConcurrentRunner. I can have the flow run successfully with only 100 mapped tasks, but I have flows in Prefect 1.0 that are up to 45k mapped tasks.
z
Zanie
10/26/2022, 6:05 PMCan you enable debug logs so we get a traceback for the crash?
:gratitude-thank-you: 1
What are you using for a backing server as well?
t
Tim Enders
10/26/2022, 6:05 PMI am connecting to my cloud API
but running locally
How would I enable DEBUG for the run... I am running the flow with
python path/to/my/file.pyz
Zanie
10/26/2022, 6:08 PMPREFECT_DEBUG_MODE=1 or PREFECT_LOGGING_LEVEL=DEBUG
(environment variables, or
prefect config set SETTING=valuet
Tim Enders
10/26/2022, 6:58 PMGot a crash result out of there. I think this is what you are looking for. I will attach it as a text file
Crash file. I have the whole stack trace, but it was emitting data so I don't really want to share it.
z
Zanie
10/26/2022, 7:16 PMThanks! What client version are you using?
We’ve made some recent changes to attempt to address this kind of connection error.
t
Tim Enders
10/26/2022, 7:18 PM2.6.4, I think that is up to date?
z
Zanie
10/26/2022, 8:04 PMYeah I’ll loop in some relevant parties. cc @Zach Angell / @Jake Kaplan
Just to check, you’re using Prefect Cloud without a proxy?
j
Jake Kaplan
10/26/2022, 8:07 PMTaking a look
:thank-you: 1
t
Tim Enders
10/27/2022, 2:04 PMCorrect on the no proxy question
j
Jake Kaplan
10/27/2022, 9:07 PMSorry for the late follow up, I wasn't able to identify any errors in Cloud related to this. As best I can we never actually got the bulk of the requests and it looks like the errors seem to be socket based on the client side. Is there anything else you can tell us about your setup?
t
Tim Enders
10/27/2022, 9:09 PMIt is just my laptop with a Fiber connection (which is newish) hitting the cloud API. I wonder if my ISP is upset about the outgoing requests? 🤔
I have upgraded to 2.6.5 today and things seem more stable? I haven't been able to test much today. I expect to do more tomorrow
2.6.5 seems to be much more stable on the API. I am not running into any issues with my larger mapped tasks runs. Thank you all for the great work you are doing on this product!
z
Zanie
10/28/2022, 4:37 PMYou're welcome! We appreciate all the feedback.