https://prefect.io logo
a

Alex Litvinov

07/07/2023, 5:39 AM
Hi everyone! I hope it's a correct place to ask. So, I have the following situation. I use prefect block to store google service account key (in json). But the problem is that the downstream code that uses it requires a path to file (
langchain.document_loaders.GoogleDriveLoader
for that matters) So I cannot just load the block and pass it directly The workaround that I'm using is to 1. load the block 2. get the secret value 3. write it to a temp file 4. use it
Copy code
temp_creds = tempfile.NamedTemporaryFile()
    creds_dict = GcpCredentials.load("google-drive-creds").service_account_info.get_secret_value()
    with open(temp_creds.name, 'w') as f_out:
        json.dump(creds_dict, f_out)
    loader = GoogleDriveLoader(service_account_key=Path(temp_creds.name),
                               document_ids=document_ids)
    raw_docs = loader.load()
    temp_creds.close()
To me this solution seems extremely ugly. And my question is - is there a way to do it better?
n

Nate

07/07/2023, 3:29 PM
hmm this strikes me as a restriction of the
GoogleDriveLoader
class, since it requires you pass a path to the credentials file instead of also accepting the value i might use a context manager for the temp file
Copy code
creds_dict = GcpCredentials.load("google-drive-creds").service_account_info.get_secret_value()

with tempfile.NamedTemporaryFile(mode='w+') as temp_creds:
    json.dump(creds_dict, temp_creds)
    loader = GoogleDriveLoader(service_account_key=Path(temp_creds.name), document_ids=document_ids)

raw_docs = loader.load()
but is there something on the prefect side you'd like to see here?
a

Alex Litvinov

07/09/2023, 8:02 AM
hmm this strikes me as a restriction of the
GoogleDriveLoader
class
I agree
but is there something on the prefect side you'd like to see here?
not really, I think prefect is doing perfect in this instance 😀 just wanted to ask the community in case someone stumbled upon the same problem and found a more elegant solution.
i might use a context manager for the temp file
yup! makes sense Thanks @Nate!!!
n

Nate

07/10/2023, 1:45 AM
cool! sounds good also for AI tooling, you might be in interested https://github.com/PrefectHQ/marvin its how @Marvin works 🙂 we dont have all the Loaders / vectorstore support but lots of nifty tools and abstractions for working with LLMs we're releasing a big rewrite very soon, you can try it out with
Copy code
❯ pip install git+<https://github.com/PrefectHQ/marvin.git>
a

Alex Litvinov

07/10/2023, 5:56 AM
Wow! looks impressive! I'll certainly give it a try!