https://prefect.io logo
Join Slack
Powered by
Hi all! I was curious if there is anything in the ...
# ask-community
d
Hi all! I was curious if there is anything in the works for creating add-ons to the prefect_azure collection for interacting/accessing Azure Key Vaults. I know I can go ahead and create it myself, though I didn't want to reinvent the wheel if it was forthcoming. 🙂
✅ 1
r
I don't believe we have anything works for it, but it sounds like something we'd be open to adding if you'd like to create an issue for it in the repository. I could see a wrapper around the Key Vault Python library fitting well into the prefect-azure collection.
👍 1
d
I created a block to connect to AzureKeyVault and registered it successfully to Prefect Cloud. However when I tried to deploy it through our CI/CD workflow, I get the following error when extracting the flow to build the deployment:
Copy code
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "src/flows/alphavantage.py", line 6, in <module>
    from __prefect_loader__ import AzureKeyVaultCredentials
ImportError: cannot import name 'AzureKeyVaultCredentials' from '__prefect_loader__' (src/flows/alphavantage.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/deploy.py", line 69, in <module>
    flow = load_flow_from_script(path=config['flow_path'], flow_name=config['flow_name'].replace('_', '-'))
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 730, in load_flow_from_script
    The script will be written to a temporary local file path so errors can refer
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 709, in load_flows_from_script
    Args:
  File "/usr/local/lib/python3.8/site-packages/prefect/context.py", line 422, in registry_from_script
    load_script_as_module(path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/alphavantage.py' encountered an exception
the code in my flow to load the block was taken directly from the cloud definition:
Copy code
from __prefect_loader__ import AzureKeyVaultCredentials

azure_key_vault_credentials_block = AzureKeyVaultCredentials.load("ai-enterprise-ops")
What could be preventing it from properly conferring with the cloud to get the block?
@Ryan Peden forgot to tag you.
r
Hi David, you'll need to install the module in your CI/CD environment if you want to use it there (or include the block file alongside your flows and then import it from the file). Registering the block gets it to show up in the UI and lets you configure it there, but Cloud doesn't keep a copy of the code - we sort of treat it like your flow code as part of the hybrid model. If there's already a copy of the block code in the CI/CD environment, then I believe you'll need to import it from the file or module it resides in and not 
__prefect_loader__
. I think the example code you see in the screenshot that uses 
__prefect_loader__
is a side effect of registering the block using 
prefect block register -f
and might not be what we want to show there, so I'm glad you posted that. I'll check with a few of my colleagues and see if we need to update that (and perhaps add to the docs to provide more information on creating blocks)
d
Thanks for the feedback and that certainly makes sense. The code for the block is currently part of the repo so I think doing a straight up import should work. I'll post if I do run into any other issues
@Ryan Peden well, changing the import to the file which is in the same directory as the flow, results in a module not found error:
Copy code
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "src/flows/alphavantage.py", line 5, in <module>
    from src.flows.azure_kv import AzureKeyVaultCredentials
ModuleNotFoundError: No module named 'src.flows'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/deploy.py", line 69, in <module>
    flow = load_flow_from_script(path=config['flow_path'], flow_name=config['flow_name'].replace('_', '-'))
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 730, in load_flow_from_script
    The script will be written to a temporary local file path so errors can refer
  File "/usr/local/lib/python3.8/site-packages/prefect/flows.py", line 709, in load_flows_from_script
    Args:
  File "/usr/local/lib/python3.8/site-packages/prefect/context.py", line 422, in registry_from_script
    load_script_as_module(path)
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/importtools.py", line 156, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'src/flows/alphavantage.py' encountered an exception
Do you have insight as to why that could be the case?
j
I would try 
from .flows.azure_kv import AzureKeyVaultCredentials
or `from flows.azure_kv import AzureKeyVaultCredentials``
d
Neither of the those path changes worked. I presume it has something to do with not being able to find the module in the GitHub Storage / filesystems object.
r
Are 
alphavantage.py
and 
azure_kv.py
are in the same directory in the repo? If so, would the import in 
alphavantage.py
just be 
from .azure_kv import AzureKeyVaultCredentials
? I tries this in my dev environment using 
load_script_as_module
the same way Prefect does and my flow loaded and ran as expected
d
@Ryan Peden hate to dig up a thread here, but I went a different route to work around the import issue I kept running into. I'm working on this again and need a resolution. So I tried the 
from .tasks.whatever import pck
you suggested and I receive this error when using the 
load_script_as_module
function:
Copy code
Extracting flow from /src/flows/maas_transform.py
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./src/flows/maas_transform.py", line 5, in <module>
    from .tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ModuleNotFoundError: No module named '__prefect_loader__.tasks'
I'm not sure why it is defaulting from the 
__prefect_loader.tasks
instead of from the 
src
of the github storage block I created above the load script call. Is this a bug or am I missing something here? Btw my folder structure is as follows:
Copy code
src
|_flows
|     |_flow_code
|_tasks
      |_task_code
Forgot to mention, I also tried this import and it also failed: 
from ..tasks.whatever import pck
with this error:
Copy code
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./src/flows/maas_transform.py", line 5, in <module>
    from ..tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ValueError: attempted relative import beyond top-level package
@Ryan Peden forgot to mention I also tried 
import src.task.whatever import pck
as well with this error:
Copy code
Extracting flow from /src/flows/maas_transform.py
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./src/flows/maas_transform.py", line 5, in <module>
    from src.tasks.databricks import get_databricks_creds, get_databricks_task_settings, log_jobs_runs_status
ModuleNotFoundError: No module named 'src.tasks'
r
Thanks for the update! Seeing the folder structure helps. I'll test a couple of things, but I'm not sure Python provides a really clean way to import something from sibling directories, i.e. importing a file src/tasks into a file in src/flows. This sort of highlights the issue. Per that link, adding 
__init__.py
to the 
src
, 
flows
, and 
tasks
directories to turn them into modules might make the 
from ..tasks.whatever import pck
import syntax works. Modifying 
sys.path
might end up being easier, though.
d
Thanks for the response. I forgot to mention I have an 
___init___.py
file in each directory (src, flows, & tasks) already. But I'll see if experimenting with the 
sys.path
helps. It's just strange to me if I run the flow locally in my dev container, it has no issues finding the modules. It's in the CI/CD process it struggles with finding the modules within the Github storage block. Is there something perhaps I'm missing to make that possible?
r
Thanks - it's good to know you've got the 
__init__.py
files. It seems like there are some edge cases around the way Prefect imports the script and I think you're the first person who has encountered them. I'm going to try to document a working example of how to make this work. One thing that should work would be making your modules pip installable and then installing their directories with 
pip install -e
but that adds a lot of overhead to what you're trying to do.
👍 1
7 Views
Open in Slack
PreviousNext
Powered by