I've been working to keep exception and log data in our environment instead of going to Prefect Cloud. Exceptions were pretty easily handled with a decorator to catch the exceptions, store them locally and then raise an exception that has the URI of the saved exception. I have a small app that pulls down the log from the cloud and builds a simple web page that lets you click to get the full exception from the local store. I wanted to do the same thing with logging. Log the message to a local file and update the message to Prefect Cloud with the URI of the locally stored log message. I can add the handler to log to a local file and I can update the formatter on the prefect logger to accomplish all of this when I have control of the application and call flow.run(). I'm having problems figuring out how to get the little stub of code that makes these change to the logging configuration to run with the agent picks up the flow from Prefect Cloud and runs the flow. Is there a place to put this code that the Agent will run it before the flow is kicked off? I guess as a last resort I can add a set task at the beginning of the flow to make this configuration change.

Nate - this is such a cool idea and one that I hadn’t considered before. To borrow Cloud’s existing vocabulary, you’ve essentially described the equivalent of a

ResultHandler

for logs: a way to ship the sanitized log to Cloud and recover the full log locally. We’ve been operating on the basis that logs in Cloud are either full opt-in or opt-out, but now that you’ve described this pattern I’m extremely interested in exploring it as a first-class middle ground.

Good point on the "same memory space". I run in Dask so that won't work. I guess I'll go back to me other plan which is to add it to the decorator that I'm using to capture and clean up the exceptions. It just feels funny to check and update the logging configuration in the decorator before each task function is actually called.

I agree with you - we’re going to take your use case into consideration and see how we can expose better hooks. Pragmatically, we generally assume each task is running in a separate process, so checking the config is “correct,” but we don’t want you to have to write code that isn’t nice and DRY

Is this something that you would want set on the process where the flow runs? Environments allow for

on_start

and

on_exit

callbacks which you may provide functions to run prior to the flow run and after it has finished. https://docs.prefect.io/cloud/execution/overview.html#environment-callbacks For example the

on_start

callback could perform your code before the flow starts however it won’t happen prior to the agent deploying it.

I have to step out for a little bit. I'll take a look at that and let you know. Right now I think it applies to the entire flow.

I agree, and glad you’re all set! It’ll be on our roadmap.

I've got a little bit more exploration to do on the Exception one as well, but I think that could also follow a similar pattern.

Hey @Nate Atkins! Which environment class are you using and would you mind sharing your logic for reconfiguring the logger within

on_start

?

I've been using the DaskExcutor and whatever it defaults to in a RemoteEnvironment. It got a little more complicated than I wanted for a proof of concept. I could also see other issues as mentioned in the PIN on environment callbacks working slightly differently. I also found that CloudHandler doesn't call a formatter so where I had the logic to add the URI there it never got called.

Gotcha gotcha, so I think what’s going on is that when dask submits work to a subprocess the logger is reconfigured from scratch. I believe we’ll need to bake in a hook into the prefect code that allows you to specify your own formatters so they are configured each time the logger is. I would expect that if you use the

LocalExecutor

this will work as you expect — would you be willing to open a Feature Request issue on GitHub for this? I think it’s a really great idea

Yes, I'm working to get some code together that demonstrates what can be done for both logging and exceptions. They won't be a proposal for how it should be implemented, but something to anchor a discussion around and flesh out some requirements. I should be able to get something open in the next day or two.

yea that sounds great thank you!!

Here is the one for Local Log Storage. https://github.com/PrefectHQ/prefect/issues/2038

wow nice thank you @Nate Atkins!