https://prefect.io logo
Powered by
Title
k

Kevin Mullins

04/13/2022, 5:02 PM
I’m starting to use 
AzureResult
to store task results for my flows. I’ve already got separate storage accounts per environment (dev, qa, prod). I’m curious if it would be recommended to use separate containers for different Prefect projects and/or flows or if it is ok to just store all the results in the same blob container for the environment. My hesitation for further separation is it appears that the AzureResult requires the container to already exist, so I would need to orchestrate something creating containers for each Prefect project. Not a big deal but just trying to get a feel for good practices. Any thoughts appreciated.
k

Kevin Kho

04/13/2022, 5:08 PM
Really not sure but I think using the same container slightly defeats the purpose of environment separation right? Because it’s not entirely isolated?
But honestly I think it’s fine and I could live combining all
k

Kevin Mullins

04/13/2022, 5:09 PM
Each environment has it’s own account so in turn their own containers, something like this: 
storage-account-prefect-dev
  prefect_results
storage-account-prefect-qa
  prefect_results
etc...
Just wondering it the one “prefect_results” container would be better or worse than something like separate “prefect_project_1_results” and “prefect_project_2_results” containers
k

Kevin Kho

04/13/2022, 5:13 PM
Ahhh I see what you mean now. That looks like it should be fine already and you can template the paths inside the big 
prefect_results
folder. You could pull the project from the API and template the location
:upvote: 1
k

Kevin Mullins

04/13/2022, 5:14 PM
Awesome, I like that approach and think it provides enough flexibility. Thanks!
This worked out pretty well for me. I already have the project and flow names available without a query: 
# generic function that can be used by all flows, in shared lib
def azure_result_location_formatter(project_name: str, flow_name: str) -> str:
    prefix = f"{sanitize_object_name(project_name)}/{sanitize_object_name(flow_name)}"
    date = pendulum.now("utc").format("Y/M/D")  # type: ignore
    location = f"{prefix}/{date}/{uuid.uuid4()}.prefect_result"
    return location

# flow specific formatter
flow_result_location_formatter = partial(azure_result_location_formatter, PROJECT_NAME, FLOW_NAME)

with Flow(name=FLOW_NAME, result=AzureResult("results", location=flow_result_location_formatter)) as flow:
    ...
k

Kevin Kho

04/13/2022, 5:54 PM
Ah I see. Yeah that’s easy then!
5 Views
#prefect-communityJoin Slack
Powered by