Hi folks! I've been poking around a bunch with Prefect for the last couple months, and over that time I've written a few utilities/extensions to help Prefect work better for my particular data and modeling workflows. In case anyone else may be able to benefit I've pulled them together into a small package — https://pypi.org/project/prefect-ds/. Please feel free to take a look 🙂

This is awesome, thanks for sharing it! If you’re game, we can look at bringing some of it directly into Prefect, there’s a lot of good ideas here.

This is really awesome @Andrew Schechtman-Rook! There are a lot of things here that I think could be incorporated almost as-is into Prefect, and some other things that we could tweak to include if you’d be interested in working together on it

yeah, ideally I'd love to see these things in prefect — I had to do some unpleasant contortions to minimize modifications to the Prefect classes to hopefully keep incompatibilities from cropping up too much, but I wasn't sure if y'all would be interested in the directions I was going in

Anyone who’s implemented a custom

Result

class definitely knows what they’re doing, trust me

awesome; yea this all seems very in-scope for the core library, and the

checkpoint_handler

is very similar to a feature we have been discussing internally so I’d love to work with you on it

lol thanks

actually i think the simplest thing could be the pandas result handler

happy to do whatever, but I'd probably want to try for the easiest, lowest-hanging fruit first

as far as I can tell, the only thing we’d need to do to introduce it in the core library is: - decide how to handle the pandas dependency (maybe as an extra) - create a serializer for it (e.g., https://github.com/PrefectHQ/prefect/blob/master/src/prefect/serialization/result_handlers.py)

would you want the handler with the full filepath specification and string formatting support, or more like the

LocalResultHandler

where the user only specifies the directory?

Honestly in this case, my bias is to include it using the configuration that you found useful for your own work because chances are someone else will find it beneficial

ok, I'll start with that and we can adjust as needed

For the other more intricate features I’ll do some more review and contact you outside of this thread for how we can work on bringing them in

sounds good

haha yea no worries!