Channels
#prefect-community
Title
# prefect-community
b
Brett Naul
04/15/2022, 1:06 PMwe're still having issues with logs not loading, sounds like there was an incident yesterday but prefect.status.io says it should be resolved now? any known issues that are still being worked on?
z
Zanie
04/15/2022, 1:29 PM
I've pinged some people
We'll get back to you soon
b
Brett Naul
04/15/2022, 1:49 PMthanks!
uh oh, am hearing that maybe the incident was our fault........... will try to see what triggered the deluge
@Zanie @Chris White if you can show any examples of the logs that were being pushed that would be helpful, I see some periods of high-ish volume but nothing stands out immediately. and since I can't view the logs from Prefect I am sort of guessing
c
Chris White
04/15/2022, 5:06 PM
Hey @Brett Naul we've made some changes on our end and should be able to get this sorted out today; I'll try to pull some example logs for oyu in the meantime
b
Brett Naul
04/18/2022, 4:37 PM@Zanie @Chris White our logs seem to still be disabled, any ETA on a fix and/or more info about what the spammy logs looked like?
c
Chris White
04/18/2022, 5:49 PM
@Brett Naul you should be unblocked now - can you confirm?
b
Brett Naul
04/18/2022, 6:01 PMlooking good now thanks! still feeling a bit uneasy though not knowing what went wrong, if you have a chance to pull a sample of the logs that would help us make sure it doesn't happen again
c
Chris White
04/18/2022, 6:25 PM
yea for sure, I'll follow up in a DM
b
Brett Naul
05/23/2022, 5:49 PMlooks like we might be in log purgatory again, @Zanie or someone else can you confirm?
c
Chris White
05/24/2022, 5:18 PM
Correct - shortly after we re-enabled logging your tenant began shipping around 1 million logs per minute; I'm going to implement a UI alert so you can definitively know when logs get disabled on your tenant
b
Brett Naul
05/24/2022, 6:20 PM😞
@Chris White do you have an example? is it related to statsd or datadog? I believe that's what was causing the problem for us but not positive
c
Chris White
05/24/2022, 6:24 PM
I can dig something up but it'll be a while (most likely tomorrow)
b
Brett Naul
05/24/2022, 6:28 PMall right, I'm fairly confident that's what it was so no worries
10 Views