https://prefect.io
Join Slack
hey all! is there a way to get Prefect to Pickle flows including dependencies?
# prefect-server
c
hey all! is there a way to get Prefect to Pickle flows including dependencies?
k
Not at the moment. Might be possible in the future due to this 
cloudpickle
change
But actually for Orion the goal this quarter is to make packaging easier to do in general whatever that means (pickle or not) such that we can handle more of that for the users
We do hear that a lot “why can’t Prefect install my Github repo for me?”
c
yea i keep hitting walls running containers and Flows that are like
Copy code
from something import thing

with Flow(...):
   thing()
then i get 
No module named 'something'
all the time and am hacking the Docker containers like mad to get the 
something
installed using 
pip install -e something
if you have any workarounds, that'd be amazing!
k
Clone and pip install the repo as an ENTRYPOINT of your container
c
Thats a good idea, but a couple downsides i can imagine with that: • super slow flow startup time for a flow? • need to inject a SSH key to clone a private repo, somewhere?
does prefect have the ability to augment/customize to the pickling system? could i "tell it what else to pickle" when running register flows?
k
Current Prefect does not have that
c
😢
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by