https://prefect.io logo
#prefect-community
Title
# prefect-community
j

jack

06/23/2022, 1:31 PM
How do I configure a flow to upload local python modules to S3? For example my directory structure looks like this:
Copy code
project/
    flow.py
    scripts/
        util.py
And in flow.py, there is this import statement:
Copy code
from scripts.util import Util
Also in flow.py, using this storage:
Copy code
with Flow('some-name', storage=S3(bucket='some-bucket')):
    ...
But when run from the cloud, it complains
No module named "scripts.util"
k

Kevin Kho

06/23/2022, 1:35 PM
If local agent, you can specify a working directory like the third example here. If you use a containerized environment, you really need to package and install them. You need the files available in the container and available in the Python path. None of the storages can inherently install these additional files. I have a packaging article here if it helps. Could we trim the parent message a bit when you get the chance? I think we don’t need the last error
Actually there is work done in 2.0 to make it simpler.
j

jack

06/23/2022, 1:56 PM
Using a local agent
k

Kevin Kho

06/23/2022, 1:57 PM
You can do this then:
Copy code
flow.run_config = LocalRun(working_dir="/path/to/working-directory")
Or start the agent in the place that has access to those
j

jack

06/23/2022, 2:08 PM
Do you have any other ideas? When registering a flow, I like that everything in the flow is snapshotted. Meaning even if I change the file on disk, what's in the cloud will keep running as it was when the flow was registered. I would prefer to keep that pattern. And I would like to keep code organized into modules. The modules are not used in any other projects, so it seems overkill to package them.
With other projects we have containerized the work, but this project calls a windows executable so that rules out docker.
k

Kevin Kho

06/23/2022, 2:14 PM
We don’t hold code like that so we can’t rollback versions either. I think Prefect 2.0 might eventually solve this because you can pin to a git branch/commit for a deployment but that hasn’t been rolled out yet
j

jack

06/23/2022, 2:26 PM
Okay. Still thinking on this...
k

Kevin Kho

06/23/2022, 2:29 PM
I think your options are either Docker or Git. The current Git storage does let you pin to a commit though I think
🤔 1
But the local files will need to use the same git commit too which might be manual i don’t know, or at the very least tricky to execute