Hi all I was curious if there is anything in the works for c

Question

Hi all I was curious if there is anything in the works for creating add ons to the prefect azure collection for interacting accessing Azure Key Vaults I know I can go ahead and create it myself though I didn t want to reinvent the wheel if it was forthcoming slightly smiling face

Ryan Peden · Answer

I don t believe we have anything works for it but it sounds like something we d be open to adding if you d like to create an issue for it in the repository I could see a wrapper around the <https learn microsoft com en us azure key vault secrets quick create python tabs=azure cli|Key Vault Python library> fitting well into the prefect azure collection

David Beck · Answer

< Ryan Peden>

David Beck · Answer

I created a block to connect to AzureKeyVault and registered it successfully to Prefect Cloud However when I tried to deploy it through our CI CD workflow I get the following error when extracting the flow to build the deployment ```Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File src flows alphavantage py line 6 in lt module gt from prefect loader import AzureKeyVaultCredentials ImportError cannot import name AzureKeyVaultCredentials from prefect loader src flows alphavantage py The above exception was the direct cause of the following exception Traceback most recent call last File deploy py line 69 in lt module gt flow = load flow from script path=config flow path flow name=config flow name replace File usr local lib python3 8 site packages prefect flows py line 730 in load flow from script The script will be written to a temporary local file path so errors can refer File usr local lib python3 8 site packages prefect flows py line 709 in load flows from script Args File usr local lib python3 8 site packages prefect context py line 422 in registry from script load script as module path File usr local lib python3 8 site packages prefect utilities importtools py line 156 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at src flows alphavantage py encountered an exception``` the code in my flow to load the block was taken directly from the cloud definition ```from prefect loader import AzureKeyVaultCredentials azure key vault credentials block = AzureKeyVaultCredentials load ai enterprise ops ``` What could be preventing it from properly conferring with the cloud to get the block

David Beck · Answer

< Ryan Peden> forgot to tag you

Ryan Peden · Answer

Hi David you ll need to install the module in your CI CD environment if you want to use it there or include the block file alongside your flows and then import it from the file Registering the block gets it to show up in the UI and lets you configure it there but Cloud doesn t keep a copy of the code we sort of treat it like your flow code as part of the hybrid model If there s already a copy of the block code in the CI CD environment then I believe you ll need to import it from the file or module it resides in and not ` prefect loader ` I think the example code you see in the screenshot that uses ` prefect loader ` is a side effect of registering the block using `prefect block register f` and might not be what we want to show there so I m glad you posted that I ll check with a few of my colleagues and see if we need to update that and perhaps add to the docs to provide more information on creating blocks

David Beck · Answer

Thanks for the feedback and that certainly makes sense The code for the block is currently part of the repo so I think doing a straight up import should work I ll post if I do run into any other issues

David Beck · Answer

< Ryan Peden> well changing the import to the file which is in the same directory as the flow results in a module not found error ```Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File src flows alphavantage py line 5 in lt module gt from src flows azure kv import AzureKeyVaultCredentials ModuleNotFoundError No module named src flows The above exception was the direct cause of the following exception Traceback most recent call last File deploy py line 69 in lt module gt flow = load flow from script path=config flow path flow name=config flow name replace File usr local lib python3 8 site packages prefect flows py line 730 in load flow from script The script will be written to a temporary local file path so errors can refer File usr local lib python3 8 site packages prefect flows py line 709 in load flows from script Args File usr local lib python3 8 site packages prefect context py line 422 in registry from script load script as module path File usr local lib python3 8 site packages prefect utilities importtools py line 156 in load script as module raise ScriptError user exc=exc path=path from exc prefect exceptions ScriptError Script at src flows alphavantage py encountered an exception``` Do you have insight as to why that could be the case

Jarvis Stubblefield · Answer

I would try `from flows azure kv import AzureKeyVaultCredentials` or `from flows azure kv import AzureKeyVaultCredentials``

David Beck · Answer

Neither of the those path changes worked I presume it has something to do with not being able to find the module in the GitHub Storage filesystems object

Ryan Peden · Answer

Are `alphavantage py` and `azure kv py` are in the same directory in the repo If so would the import in `alphavantage py` just be `from azure kv import AzureKeyVaultCredentials` I tries this in my dev environment using `load script as module` the same way Prefect does and my flow loaded and ran as expected

David Beck · Answer

< Ryan Peden> hate to dig up a thread here but I went a different route to work around the import issue I kept running into I m working on this again and need a resolution So I tried the `from tasks whatever import pck` you suggested and I receive this error when using the `load script as module` function ```Extracting flow from src flows maas transform py Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File src flows maas transform py line 5 in lt module gt from tasks databricks import get databricks creds get databricks task settings log jobs runs status ModuleNotFoundError No module named prefect loader tasks ``` I m not sure why it is defaulting from the ` prefect loader tasks` instead of from the `src` of the github storage block I created above the load script call Is this a bug or am I missing something here Btw my folder structure is as follows ```src | flows | | flow code | tasks | task code``` Forgot to mention I also tried this import and it also failed `from tasks whatever import pck` with this error ```Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File src flows maas transform py line 5 in lt module gt from tasks databricks import get databricks creds get databricks task settings log jobs runs status ValueError attempted relative import beyond top level package```

David Beck · Answer

< Ryan Peden> forgot to mention I also tried `import src task whatever import pck` as well with this error ```Extracting flow from src flows maas transform py Traceback most recent call last File lt frozen importlib bootstrap external gt line 843 in exec module File lt frozen importlib bootstrap gt line 219 in call with frames removed File src flows maas transform py line 5 in lt module gt from src tasks databricks import get databricks creds get databricks task settings log jobs runs status ModuleNotFoundError No module named src tasks ```

Ryan Peden · Answer

Thanks for the update Seeing the folder structure helps I ll test a couple of things but I m not sure Python provides a really clean way to import something from sibling directories i e importing a file src tasks into a file in src flows <https stackoverflow com questions 10272879 how do i import a python script from a sibling directory|This> sort of highlights the issue Per that link adding ` init py` to the `src` `flows` and `tasks` directories to turn them into modules might make the `from tasks whatever import pck` import syntax works Modifying `sys path` might end up being easier though

David Beck · Answer

Thanks for the response I forgot to mention I have an ` init py` file in each directory src flows amp tasks already But I ll see if experimenting with the `sys path` helps It s just strange to me if I run the flow locally in my dev container it has no issues finding the modules It s in the CI CD process it struggles with finding the modules within the Github storage block Is there something perhaps I m missing to make that possible

Ryan Peden · Answer

Thanks it s good to know you ve got the ` init py` files It seems like there are some edge cases around the way Prefect imports the script and I think you re the first person who has encountered them I m going to try to document a working example of how to make this work One thing that should work would be making your modules pip installable and then installing their directories with `pip install e` but that adds a lot of overhead to what you re trying to do