Hi folks! I've been poking around a bunch with Pre...
# prefect-community
a
Hi folks! I've been poking around a bunch with Prefect for the last couple months, and over that time I've written a few utilities/extensions to help Prefect work better for my particular data and modeling workflows. In case anyone else may be able to benefit I've pulled them together into a small package — https://pypi.org/project/prefect-ds/. Please feel free to take a look 🙂
👊 5
marvin 4
❤️ 7
🚀 7
👏 5
j
This is awesome, thanks for sharing it! If you’re game, we can look at bringing some of it directly into Prefect, there’s a lot of good ideas here.
upvote 2
c
This is really awesome @Andrew Schechtman-Rook! There are a lot of things here that I think could be incorporated almost as-is into Prefect, and some other things that we could tweak to include if you’d be interested in working together on it
upvote 2
a
yeah, ideally I'd love to see these things in prefect — I had to do some unpleasant contortions to minimize modifications to the Prefect classes to hopefully keep incompatibilities from cropping up too much, but I wasn't sure if y'all would be interested in the directions I was going in
my spare time is hit-and-miss, but I'm happy to help on it as I have time
j
Anyone who’s implemented a custom
Result
class definitely knows what they’re doing, trust me
c
awesome; yea this all seems very in-scope for the core library, and the
checkpoint_handler
is very similar to a feature we have been discussing internally so I’d love to work with you on it
a
lol thanks
sure, do you think the
checkpoint_handler
is the best thing to go after first?
c
actually i think the simplest thing could be the pandas result handler
a
happy to do whatever, but I'd probably want to try for the easiest, lowest-hanging fruit first
c
as far as I can tell, the only thing we’d need to do to introduce it in the core library is: - decide how to handle the pandas dependency (maybe as an extra) - create a serializer for it (e.g., https://github.com/PrefectHQ/prefect/blob/master/src/prefect/serialization/result_handlers.py)
a
would you want the handler with the full filepath specification and string formatting support, or more like the
LocalResultHandler
where the user only specifies the directory?
(also, feel free to switch to DMs, issue on a repo, email, or whatever is convenient for you)
👍 1
c
Honestly in this case, my bias is to include it using the configuration that you found useful for your own work because chances are someone else will find it beneficial
a
ok, I'll start with that and we can adjust as needed
👍 1
c
For the other more intricate features I’ll do some more review and contact you outside of this thread for how we can work on bringing them in
a
sounds good
I'll start working on it next time I have a free moment, we'll see when that is 😛
c
haha yea no worries!
👍 1