Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

we're still having issues with logs not loading, sounds like there was an incident yesterday but <http://prefect.status.io|prefect.status.io> says it should be resolved now? any known issues that are still being worked on?

uh oh, am hearing that maybe the incident was our fault........... will try to see what triggered the deluge

<@U01CEUST9B5> <@UKNSNMUE6> if you can show any examples of the logs that were being pushed that would be helpful, I see some periods of high-ish volume but nothing stands out immediately. and since I can't view the logs from Prefect I am sort of guessing

Hey <@UL455076D> we've made some changes on our end and should be able to get this sorted out today; I'll try to pull some example logs for oyu in the meantime

<@U01CEUST9B5> <@UKNSNMUE6> our logs seem to still be disabled, any ETA on a fix and/or more info about what the spammy logs looked like?

<@UL455076D> you should be unblocked now - can you confirm?

looking good now thanks! still feeling a bit uneasy though not knowing what went wrong, if you have a chance to pull a sample of the logs that would help us make sure it doesn't happen again

looks like we might be in log purgatory again, <@U01CEUST9B5> or someone else can you confirm?

Correct - shortly after we re-enabled logging your tenant began shipping around 1 million logs per minute; I'm going to implement a UI alert so you can definitively know when logs get disabled on your tenant

<@UKNSNMUE6> do you have an example? is it related to statsd or datadog? I believe that's what was causing the problem for us but not positive

I can dig something up but it'll be a while (most likely tomorrow)

all right, I'm fairly confident that's what it was so no worries