https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Mary Clair Thompson

06/13/2022, 7:08 PM
Apologies if this has already been answered--but in the 'old' Prefect we started up agents with the -p flag so that we can point them to local modules that our flows needed to import. This functionality seems to have gone away with Orion. Is there some analogous way to indicate to the agent the location of local files that need to be imported? For reference, the short stack trace I'm getting is
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/tmp/flow-script-calculate-blob-size4cemrmn_.py", line 1, in <module>
ModuleNotFoundError: No module named 'local_module'
1
k

Kevin Kho

06/13/2022, 7:10 PM
Can I see what the DeploymentSpec looks like in this case?
m

Mary Clair Thompson

06/13/2022, 7:13 PM
Copy code
DeploymentSpec(flow_location=/path/to/flow.py,
               name="blob-size-calc-test-deployment",
               schedule=CronSchedule(cron="30 03 * * *"),
               flow_name="calculate-blob-size",
               parameters={"st_acct": "oitdatatransferdev", "container_name": "prefect-orion-test-storage", "config_path": /path/to/configs.yml},
               tags=["test"])
k

Kevin Kho

06/13/2022, 7:15 PM
What does this end up running on? Subprocess, Docker or Kubernetes?
m

Mary Clair Thompson

06/13/2022, 7:15 PM
subprocess
k

Kevin Kho

06/13/2022, 7:17 PM
I would say: 1. The agent might be running in a directory without access to those files so you can try running it from the same folder so the subprocess will have access. 2. You can add the folder to the PYTHONPATH so it can be imported from other places. 3. Similar to two, you can make it a module and
pip install
the module
💯 1
👍 1
m

Mary Clair Thompson

06/13/2022, 7:20 PM
I was hoping to avoid packaging the module...for 1 how do i determine where the agent is running?
k

Kevin Kho

06/13/2022, 7:22 PM
Just the existing directory when you did
prefect agent start work_queue_id
m

Mary Clair Thompson

06/13/2022, 7:22 PM
ah gotcha, thank you Kevin!
k

Kevin Kho

06/13/2022, 7:23 PM
Maybe it will work. Not 100% sure. For Prefect 1, that is the case.
👍 1
m

Mary Clair Thompson

06/13/2022, 7:24 PM
I'll keep you posted--about to test it.
a

Anna Geller

06/13/2022, 7:25 PM
if you really want to make it easy, you can create a conda environment and install your custom module as an editable package inside
Copy code
pip install -e .
m

Mary Clair Thompson

06/13/2022, 7:26 PM
good call, thanks Anna!
👍 1
@Kevin Kho starting the agent from the directory containing the imported file worked! thank you!
👍 1
2 Views