David Beck
11/08/2022, 8:08 PMRyan Peden
11/08/2022, 8:27 PMDavid Beck
11/08/2022, 8:37 PMTraceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "src/flows/alphavantage.py", line 6, in <module>
from __prefect_loader__ import AzureKeyVaultCredentials
ImportError: cannot import name 'AzureKeyVaultCredentials' from '__prefect_loader__' (src/flows/alphavantage.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/deploy.py", line 69, in <module>
flow = load_flow_from_script(path=config['flow_path'], flow_name=config['flow_name'].replace('_', '-'))
File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 730, in load_flow_from_script
The script will be written to a temporary local file path so errors can refer
File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 709, in load_flows_from_script
Args:
File "/usr/local/lib/python3.8/site-packages/prefect/context.py", line 422, in registry_from_script
load_script_as_module(path)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/alphavantage.py' encountered an exception
the code in my flow to load the block was taken directly from the cloud definition:
from __prefect_loader__ import AzureKeyVaultCredentials
azure_key_vault_credentials_block = AzureKeyVaultCredentials.load("ai-enterprise-ops")
What could be preventing it from properly conferring with the cloud to get the block?Ryan Peden
11/08/2022, 11:18 PM__prefect_loader__
.
I think the example code you see in the screenshot that uses __prefect_loader__
is a side effect of registering the block using prefect block register -f
and might not be what we want to show there, so I'm glad you posted that. I'll check with a few of my colleagues and see if we need to update that (and perhaps add to the docs to provide more information on creating blocks)David Beck
11/08/2022, 11:27 PMTraceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "src/flows/alphavantage.py", line 5, in <module>
from src.flows.azure_kv import AzureKeyVaultCredentials
ModuleNotFoundError: No module named 'src.flows'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/deploy.py", line 69, in <module>
flow = load_flow_from_script(path=config['flow_path'], flow_name=config['flow_name'].replace('_', '-'))
File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 730, in load_flow_from_script
The script will be written to a temporary local file path so errors can refer
File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 709, in load_flows_from_script
Args:
File "/usr/local/lib/python3.8/site-packages/prefect/context.py", line 422, in registry_from_script
load_script_as_module(path)
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/alphavantage.py' encountered an exception
Do you have insight as to why that could be the case?Jarvis Stubblefield
11/09/2022, 8:58 AMfrom .flows.azure_kv import AzureKeyVaultCredentials
or `from flows.azure_kv import AzureKeyVaultCredentials``David Beck
11/09/2022, 3:31 PMRyan Peden
11/09/2022, 3:44 PMalphavantage.py
and azure_kv.py
are in the same directory in the repo? If so, would the import in alphavantage.py
just be from .azure_kv import AzureKeyVaultCredentials
?
I tries this in my dev environment using load_script_as_module
the same way Prefect does and my flow loaded and ran as expectedDavid Beck
11/17/2022, 8:43 PMfrom .tasks.whatever import pck
you suggested and I receive this error when using the load_script_as_module
function:
Extracting flow from /src/flows/maas_transform.py
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "./src/flows/maas_transform.py", line 5, in <module>
from .tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ModuleNotFoundError: No module named '__prefect_loader__.tasks'
I'm not sure why it is defaulting from the __prefect_loader.tasks
instead of from the src
of the github storage block I created above the load script call. Is this a bug or am I missing something here?
Btw my folder structure is as follows:
src
|_flows
| |_flow_code
|_tasks
|_task_code
Forgot to mention, I also tried this import and it also failed: from ..tasks.whatever import pck
with this error:
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "./src/flows/maas_transform.py", line 5, in <module>
from ..tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ValueError: attempted relative import beyond top-level package
import src.task.whatever import pck
as well with this error:
Extracting flow from /src/flows/maas_transform.py
Traceback (most recent call last):
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "./src/flows/maas_transform.py", line 5, in <module>
from src.tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ModuleNotFoundError: No module named 'src.tasks'
Ryan Peden
11/17/2022, 11:19 PM__init__.py
to the src
, flows
, and tasks
directories to turn them into modules might make the from ..tasks.whatever import pck
import syntax works. Modifying sys.path
might end up being easier, though.David Beck
11/18/2022, 3:17 PM___init___.py
file in each directory (src, flows, & tasks) already. But I'll see if experimenting with the sys.path
helps. It's just strange to me if I run the flow locally in my dev container, it has no issues finding the modules. It's in the CI/CD process it struggles with finding the modules within the Github storage block. Is there something perhaps I'm missing to make that possible?Ryan Peden
11/18/2022, 3:23 PM__init__.py
files. It seems like there are some edge cases around the way Prefect imports the script and I think you're the first person who has encountered them. I'm going to try to document a working example of how to make this work.
One thing that should work would be making your modules pip installable and then installing their directories with pip install -e
but that adds a lot of overhead to what you're trying to do.