https://prefect.io logo
#prefect-community
Title
# prefect-community
d

David Beck

11/08/2022, 8:08 PM
Hi all! I was curious if there is anything in the works for creating add-ons to the prefect_azure collection for interacting/accessing Azure Key Vaults. I know I can go ahead and create it myself, though I didn't want to reinvent the wheel if it was forthcoming. 🙂
1
r

Ryan Peden

11/08/2022, 8:27 PM
I don't believe we have anything works for it, but it sounds like something we'd be open to adding if you'd like to create an issue for it in the repository. I could see a wrapper around the Key Vault Python library fitting well into the prefect-azure collection.
👍 1
d

David Beck

11/08/2022, 8:37 PM
I created a block to connect to AzureKeyVault and registered it successfully to Prefect Cloud. However when I tried to deploy it through our CI/CD workflow, I get the following error when extracting the flow to build the deployment:
Copy code
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "src/flows/alphavantage.py", line 6, in <module>
    from __prefect_loader__ import AzureKeyVaultCredentials
ImportError: cannot import name 'AzureKeyVaultCredentials' from '__prefect_loader__' (src/flows/alphavantage.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/deploy.py", line 69, in <module>
    flow = load_flow_from_script(path=config['flow_path'], flow_name=config['flow_name'].replace('_', '-'))
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 730, in load_flow_from_script
    The script will be written to a temporary local file path so errors can refer
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 709, in load_flows_from_script
    Args:
  File "/usr/local/lib/python3.8/site-packages/prefect/context.py", line 422, in registry_from_script
    load_script_as_module(path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/alphavantage.py' encountered an exception
the code in my flow to load the block was taken directly from the cloud definition:
Copy code
from __prefect_loader__ import AzureKeyVaultCredentials

azure_key_vault_credentials_block = AzureKeyVaultCredentials.load("ai-enterprise-ops")
What could be preventing it from properly conferring with the cloud to get the block?
@Ryan Peden forgot to tag you.
r

Ryan Peden

11/08/2022, 11:18 PM
Hi David, you'll need to install the module in your CI/CD environment if you want to use it there (or include the block file alongside your flows and then import it from the file). Registering the block gets it to show up in the UI and lets you configure it there, but Cloud doesn't keep a copy of the code - we sort of treat it like your flow code as part of the hybrid model. If there's already a copy of the block code in the CI/CD environment, then I believe you'll need to import it from the file or module it resides in and not
__prefect_loader__
. I think the example code you see in the screenshot that uses
__prefect_loader__
is a side effect of registering the block using
prefect block register -f
and might not be what we want to show there, so I'm glad you posted that. I'll check with a few of my colleagues and see if we need to update that (and perhaps add to the docs to provide more information on creating blocks)
d

David Beck

11/08/2022, 11:27 PM
Thanks for the feedback and that certainly makes sense. The code for the block is currently part of the repo so I think doing a straight up import should work. I'll post if I do run into any other issues
@Ryan Peden well, changing the import to the file which is in the same directory as the flow, results in a module not found error:
Copy code
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "src/flows/alphavantage.py", line 5, in <module>
    from src.flows.azure_kv import AzureKeyVaultCredentials
ModuleNotFoundError: No module named 'src.flows'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/deploy.py", line 69, in <module>
    flow = load_flow_from_script(path=config['flow_path'], flow_name=config['flow_name'].replace('_', '-'))
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 730, in load_flow_from_script
    The script will be written to a temporary local file path so errors can refer
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 709, in load_flows_from_script
    Args:
  File "/usr/local/lib/python3.8/site-packages/prefect/context.py", line 422, in registry_from_script
    load_script_as_module(path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/alphavantage.py' encountered an exception
Do you have insight as to why that could be the case?
j

Jarvis Stubblefield

11/09/2022, 8:58 AM
I would try
from .flows.azure_kv import AzureKeyVaultCredentials
or `from flows.azure_kv import AzureKeyVaultCredentials``
d

David Beck

11/09/2022, 3:31 PM
Neither of the those path changes worked. I presume it has something to do with not being able to find the module in the GitHub Storage / filesystems object.
r

Ryan Peden

11/09/2022, 3:44 PM
Are
alphavantage.py
and
azure_kv.py
are in the same directory in the repo? If so, would the import in
alphavantage.py
just be
from .azure_kv import AzureKeyVaultCredentials
? I tries this in my dev environment using
load_script_as_module
the same way Prefect does and my flow loaded and ran as expected
d

David Beck

11/17/2022, 8:43 PM
@Ryan Peden hate to dig up a thread here, but I went a different route to work around the import issue I kept running into. I'm working on this again and need a resolution. So I tried the
from .tasks.whatever import pck
you suggested and I receive this error when using the
load_script_as_module
function:
Copy code
Extracting flow from /src/flows/maas_transform.py
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./src/flows/maas_transform.py", line 5, in <module>
    from .tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ModuleNotFoundError: No module named '__prefect_loader__.tasks'
I'm not sure why it is defaulting from the
__prefect_loader.tasks
instead of from the
src
of the github storage block I created above the load script call. Is this a bug or am I missing something here? Btw my folder structure is as follows:
Copy code
src
|_flows
|     |_flow_code
|_tasks
      |_task_code
Forgot to mention, I also tried this import and it also failed:
from ..tasks.whatever import pck
with this error:
Copy code
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./src/flows/maas_transform.py", line 5, in <module>
    from ..tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ValueError: attempted relative import beyond top-level package
@Ryan Peden forgot to mention I also tried
import src.task.whatever import pck
as well with this error:
Copy code
Extracting flow from /src/flows/maas_transform.py
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./src/flows/maas_transform.py", line 5, in <module>
    from src.tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ModuleNotFoundError: No module named 'src.tasks'
r

Ryan Peden

11/17/2022, 11:19 PM
Thanks for the update! Seeing the folder structure helps. I'll test a couple of things, but I'm not sure Python provides a really clean way to import something from sibling directories, i.e. importing a file src/tasks into a file in src/flows. This sort of highlights the issue. Per that link, adding
__init__.py
to the
src
,
flows
, and
tasks
directories to turn them into modules might make the
from ..tasks.whatever import pck
import syntax works. Modifying
sys.path
might end up being easier, though.
d

David Beck

11/18/2022, 3:17 PM
Thanks for the response. I forgot to mention I have an
___init___.py
file in each directory (src, flows, & tasks) already. But I'll see if experimenting with the
sys.path
helps. It's just strange to me if I run the flow locally in my dev container, it has no issues finding the modules. It's in the CI/CD process it struggles with finding the modules within the Github storage block. Is there something perhaps I'm missing to make that possible?
r

Ryan Peden

11/18/2022, 3:23 PM
Thanks - it's good to know you've got the
__init__.py
files. It seems like there are some edge cases around the way Prefect imports the script and I think you're the first person who has encountered them. I'm going to try to document a working example of how to make this work. One thing that should work would be making your modules pip installable and then installing their directories with
pip install -e
but that adds a lot of overhead to what you're trying to do.
👍 1
4 Views