https://prefect.io logo
#prefect-community
Title
# prefect-community
b

Brett Naul

04/15/2022, 1:06 PM
we're still having issues with logs not loading, sounds like there was an incident yesterday but prefect.status.io says it should be resolved now? any known issues that are still being worked on?
z

Zanie

04/15/2022, 1:29 PM
I've pinged some people
We'll get back to you soon
b

Brett Naul

04/15/2022, 1:49 PM
thanks!
uh oh, am hearing that maybe the incident was our fault........... will try to see what triggered the deluge
@Zanie @Chris White if you can show any examples of the logs that were being pushed that would be helpful, I see some periods of high-ish volume but nothing stands out immediately. and since I can't view the logs from Prefect I am sort of guessing
c

Chris White

04/15/2022, 5:06 PM
Hey @Brett Naul we've made some changes on our end and should be able to get this sorted out today; I'll try to pull some example logs for oyu in the meantime
b

Brett Naul

04/18/2022, 4:37 PM
@Zanie @Chris White our logs seem to still be disabled, any ETA on a fix and/or more info about what the spammy logs looked like?
c

Chris White

04/18/2022, 5:49 PM
@Brett Naul you should be unblocked now - can you confirm?
b

Brett Naul

04/18/2022, 6:01 PM
looking good now thanks! still feeling a bit uneasy though not knowing what went wrong, if you have a chance to pull a sample of the logs that would help us make sure it doesn't happen again
c

Chris White

04/18/2022, 6:25 PM
yea for sure, I'll follow up in a DM
b

Brett Naul

05/23/2022, 5:49 PM
looks like we might be in log purgatory again, @Zanie or someone else can you confirm?
c

Chris White

05/24/2022, 5:18 PM
Correct - shortly after we re-enabled logging your tenant began shipping around 1 million logs per minute; I'm going to implement a UI alert so you can definitively know when logs get disabled on your tenant
b

Brett Naul

05/24/2022, 6:20 PM
😞
@Chris White do you have an example? is it related to statsd or datadog? I believe that's what was causing the problem for us but not positive
c

Chris White

05/24/2022, 6:24 PM
I can dig something up but it'll be a while (most likely tomorrow)
b

Brett Naul

05/24/2022, 6:28 PM
all right, I'm fairly confident that's what it was so no worries
10 Views