Channels
#prefect-community
Title
# prefect-community
m
Mary Clair Thompson
06/13/2022, 7:08 PMApologies if this has already been answered--but in the 'old' Prefect we started up agents with the -p flag so that we can point them to local modules that our flows needed to import. This functionality seems to have gone away with Orion. Is there some analogous way to indicate to the agent the location of local files that need to be imported? For reference, the short stack trace I'm getting is
Flow could not be retrieved from deployment.Traceback (most recent call last):File "/tmp/flow-script-calculate-blob-size4cemrmn_.py", line 1, in <module>ModuleNotFoundError: No module named 'local_module'✅ 1
k
Kevin Kho
06/13/2022, 7:10 PM
Can I see what the DeploymentSpec looks like in this case?
m
Mary Clair Thompson
06/13/2022, 7:13 PM Copy code
DeploymentSpec(flow_location=/path/to/flow.py,
name="blob-size-calc-test-deployment",
schedule=CronSchedule(cron="30 03 * * *"),
flow_name="calculate-blob-size",
parameters={"st_acct": "oitdatatransferdev", "container_name": "prefect-orion-test-storage", "config_path": /path/to/configs.yml},
tags=["test"])k
Kevin Kho
06/13/2022, 7:15 PM
What does this end up running on? Subprocess, Docker or Kubernetes?
m
Mary Clair Thompson
06/13/2022, 7:15 PMsubprocess
k
Kevin Kho
06/13/2022, 7:17 PM
I would say:
1. The agent might be running in a directory without access to those files so you can try running it from the same folder so the subprocess will have access.
2. You can add the folder to the PYTHONPATH so it can be imported from other places.
3. Similar to two, you can make it a module and
pip install💯 1
👍 1
m
Mary Clair Thompson
06/13/2022, 7:20 PMI was hoping to avoid packaging the module...for 1 how do i determine where the agent is running?
k
Kevin Kho
06/13/2022, 7:22 PM
Just the existing directory when you did
prefect agent start work_queue_idm
Mary Clair Thompson
06/13/2022, 7:22 PMah gotcha, thank you Kevin!
k
Kevin Kho
06/13/2022, 7:23 PM
Maybe it will work. Not 100% sure. For Prefect 1, that is the case.
👍 1
m
Mary Clair Thompson
06/13/2022, 7:24 PMI'll keep you posted--about to test it.
a
Anna Geller
06/13/2022, 7:25 PM
if you really want to make it easy, you can create a conda environment and install your custom module as an editable package inside
Copy code
pip install -e .m
Mary Clair Thompson
06/13/2022, 7:26 PMgood call, thanks Anna!
👍 1
@Kevin Kho starting the agent from the directory containing the imported file worked! thank you!
👍 1
2 Views