Flow question (sorry if this is the wrong place, w...
# ask-community
t
Flow question (sorry if this is the wrong place, wasn't sure where else it fit). If I set up a flow to use git with additional storage, as per https://docs.prefect.io/orchestration/flow_config/storage.html#loading-additional-files-with-git-storage, am I able to import other python files from that folder?
k
This is the right place. The answer is no for Python files. We’ve had people try to hack the path but I haven’t seen anyone successful attempt. In that case, you would need Docker.
t
Hmm, is there another way to import libraries? I've got some already written for interfacing with databases; possible to reuse them here in any way?
k
They just need to be available in your execution environment so you can just install them on your agent (for local agent).
Then you can specify the working directory to run your flow from
t
Hmm, is that process mapped out in the docs anywhere? If it is I'm missing it.
k
The working dir is here
t
Hey Kevin, digging into this again, the local agent approach does indeed look like it works for me for now. The documentation recommends against it in most cases though; is there the possibility of code reuse outside the local agent?
k
Could you show me where it recommends against it? It should be fine. For docker and k8s agent, it would be in the image because those create new environments to run the flow.
t
https://docs.prefect.io/orchestration/agents/local.html Right at the top: "While the local agent is fully capable of executing flows in conjunction with the Prefect API, we generally recommend using one of the other agents to help with modularity and scale."
I'd like modularity and scale and code reuse ideally. 😄
k
So I have seen people just spin up a local agent on a VM and use that in production. I think the comparison there is Kubernetes with autoscaling so that you can just get resources for your flows “on demand”. Local agent will work and your can install the stuff on there. If you want to go to Docker or Kubernetes, you definitely need to package those dependencies in a Docker container
I think there might be flexibility around this for Orion (Prefect 2.0) because
cloudpickle
released 2.0 which allows deepcopying of modules but Prefect 1.0 doesnt support this
t
Following up on this; Local Agent seems to be working to cover my current needs, and I've got a Docker possibility lined up for if I expand. Thanks!
k
You got the ports open?
t
No, still waiting on that, but in the meantime I spun up a new Linux VM and installed a GUI on it so I could at least access the GUI from itself. For the moment I'm developing against that while I wait for the ports to be opened.
k
Oh I see