I m starting to use `AzureResult` to store task results for Prefect Community #prefect-community

I’m starting to use `AzureResult` to store task re...

Kevin Mullins

04/13/2022, 5:02 PM

I’m starting to use

AzureResult

to store task results for my flows. I’ve already got separate storage accounts per environment (dev, qa, prod). I’m curious if it would be recommended to use separate containers for different Prefect projects and/or flows or if it is ok to just store all the results in the same blob container for the environment. My hesitation for further separation is it appears that the AzureResult requires the container to already exist, so I would need to orchestrate something creating containers for each Prefect project. Not a big deal but just trying to get a feel for good practices. Any thoughts appreciated.

Kevin Kho

04/13/2022, 5:08 PM

Really not sure but I think using the same container slightly defeats the purpose of environment separation right? Because it’s not entirely isolated?

Kevin Kho

04/13/2022, 5:09 PM

But honestly I think it’s fine and I could live combining all

Kevin Mullins

04/13/2022, 5:09 PM

Each environment has it’s own account so in turn their own containers, something like this:

Copy code

storage-account-prefect-dev
  prefect_results
storage-account-prefect-qa
  prefect_results
etc...

Kevin Mullins

04/13/2022, 5:10 PM

Just wondering it the one “prefect_results” container would be better or worse than something like separate “prefect_project_1_results” and “prefect_project_2_results” containers

Kevin Kho

04/13/2022, 5:13 PM

Ahhh I see what you mean now. That looks like it should be fine already and you can template the paths inside the big

prefect_results

folder. You could pull the project from the API and template the location

upvote 1

Kevin Mullins

04/13/2022, 5:14 PM

Awesome, I like that approach and think it provides enough flexibility. Thanks!

Kevin Mullins

04/13/2022, 5:44 PM

This worked out pretty well for me. I already have the project and flow names available without a query:

Copy code

# generic function that can be used by all flows, in shared lib
def azure_result_location_formatter(project_name: str, flow_name: str) -> str:
    prefix = f"{sanitize_object_name(project_name)}/{sanitize_object_name(flow_name)}"
    date = pendulum.now("utc").format("Y/M/D")  # type: ignore
    location = f"{prefix}/{date}/{uuid.uuid4()}.prefect_result"
    return location

# flow specific formatter
flow_result_location_formatter = partial(azure_result_location_formatter, PROJECT_NAME, FLOW_NAME)

with Flow(name=FLOW_NAME, result=AzureResult("results", location=flow_result_location_formatter)) as flow:
    ...

Kevin Kho

04/13/2022, 5:54 PM

Ah I see. Yeah that’s easy then!

5 Views

Open in Slack

Previous Next