Kevin Mullins
04/21/2022, 2:06 PMAKS
. I’ve been using Docker
storage for my flows and just recently started using AzureResult
in my flows for result persistence. I’ve been letting the AzureResult
use the AZURE_STORAGE_CONNECTION_STRING
environment variable which is the recommendation from the documentation and plays nicely with k8s
secrets in AKS
. Just found after some infrastructure changes that the environment variable was not being used and finally found it it was because the Docker
storage was capturing the environment variable during registration and serializing the value via cloudpickle
into the flow. I’ll be looking into using secrets; however, I wanted to suggest maybe re-visiting the documentation or process to prevent accidental credential exposures in the Docker
storage.Kevin Kho
Kevin Mullins
04/21/2022, 2:11 PMFlow
level.
def flow_based_azure_result(
project_name: str,
flow_name: str,
container_name: str = "prefect-results",
**kwargs: Any,
) -> AzureResult:
"""Create an AzureResult for a flow."""
location_formatter = partial(
azure_result_location_formatter, project_name, flow_name
)
return AzureResult(container_name, location=location_formatter, **kwargs)
I use this when constructing the flow:
with Flow(
name=FLOW_NAME,
schedule=schedule,
executor=LocalDaskExecutor(),
result=flow_based_azure_result(PROJECT_NAME, FLOW_NAME),
) as flow:
Kevin Kho
Kevin Mullins
04/21/2022, 2:31 PMinitialize_service
method like the secrets management is.
I see that it falls back to DefaultAzureCredential
. I didn’t see this documented but eventually I would like to get all my pods running with a managed identity and/or service principal for Azure stuff.Azure
storage (I’m not) since it instantiates AzureResult
internally: https://github.com/PrefectHQ/prefect/blob/0ba43607572cfbd7e88050e28f92377eab09e441/src/prefect/storage/azure.py#L59Kevin Kho
Kyle McChesney
04/21/2022, 2:47 PMdef get_some_environment_variable():
return os.environ['SOME_ENV_VAR']
instead of doing something like this (at the “top level” of the flow where its evaluated at import time):
some_environment_variable = os.environ['SOME_ENV_VAR']
Kevin Mullins
04/21/2022, 3:14 PMKevin Kho