Hey folks, having some persistent issues with the Secrets Vault. In a mapped task with 20 tasks, for...

Jacob Blanco

12/03/2021, 2:11 PM

Hey folks, having some persistent issues with the Secrets Vault. In a mapped task with 20 tasks, for some reason 1 out of the 20 mapped runs might error out complaining that Secret A is not available even though the other 19 found Secret A before and after the failing task.

Anna Geller

12/03/2021, 2:13 PM

@Jacob Blanco that’s unusual. Perhaps you could retrieve this secret only once and pass it to the mapped tasks as “unmapped” data dependency?

👍 1

Anna Geller

12/03/2021, 2:17 PM

here is an example:

Copy code

from prefect import task, Flow, unmapped
from prefect.tasks.secrets import PrefectSecret


@task
def say_hi(user_name, secret):
    print(f"Hi {user_name}")


with Flow("unmapped_serets"):
    credential = PrefectSecret("SOME_SECRET")
    list_names = ["Jacobo", "Marvin"]
    say_hi.map(list_names, secret=unmapped(credential))

Kevin Kho

12/03/2021, 2:44 PM

I think you can also try adding it to storage.

Copy code

flow.storage = SomeStorage(..., secrets=["SECRET_NAME"])

to reduce the API calls

upvote 1

Marko Herkaliuk

12/03/2021, 3:24 PM

or set retries to secret task

Copy code

secret_task = PrefectSecret(
    max_retries=5,
    retry_delay=datetime.timedelta(seconds=5),
)

Kevin Kho

12/03/2021, 3:25 PM

Hahaha that’s clever @Marko Herkaliuk

🙏 1

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.