James Ashby
09/08/2023, 6:31 PMMarvin
09/08/2023, 6:32 PMJames Ashby
09/08/2023, 7:27 PMMarvin
09/08/2023, 7:28 PMkey
in the S3Result
class. This key
is essentially a path within your S3 bucket where the results will be stored.
Here's an example of how you might do it:
python
from prefect.engine.results import S3Result
result = S3Result(
bucket="my-bucket-name",
key="results/{flow_run_name}-{task_full_name}", # This is the prefix
)
In this case, the results will be stored in a directory structure that looks like this: results/flow_run_name-task_full_name
.
You can customize the key
to suit your needs. For example, if you want all results to simply go into a results/
directory, you could do:
python
result = S3Result(
bucket="my-bucket-name",
key="results/",
)
Remember to replace "my-bucket-name"
with the name of your actual S3 bucket. I'm sure you would have figured that out on your own, but I thought I'd mention it just in case.
Once you've set this up, you can create a lifecycle rule for the results/
prefix in your S3 bucket, and your results will be automatically cleaned up like the crumbs from a hastily eaten sandwich.
You're welcome.James Ashby
09/08/2023, 7:29 PMMarvin
09/08/2023, 7:30 PMS3Result
instance to the storage
parameter of the Flow
class or decorator. This instance should have your desired key
(the prefix) set.
Here's an example of how you might do it:
python
from prefect import Flow
from prefect.engine.results import S3Result
@Flow(
name="my-flow",
result=S3Result(
bucket="my-bucket-name",
key="results/{flow_run_name}-{task_full_name}", # This is the prefix
),
)
def my_flow():
# Your flow's tasks go here...
In this case, the results of the tasks in my_flow
will be stored in a directory structure that looks like this: results/flow_run_name-task_full_name
.
You can customize the key
to suit your needs. For example, if you want all results to simply go into a results/
directory, you could do:
python
@Flow(
name="my-flow",
result=S3Result(
bucket="my_bucket_name",
key="results/",
),
)
def my_flow():
# Your flow's tasks go here...
Remember to replace "my-bucket-name"
with the name of your actual S3 bucket. I'm sure you would have figured that out on your own, but I thought I'd mention it just in case.
Once you've set this up, you can create a lifecycle rule for the results/
prefix in your S3 bucket, and your results will be automatically cleaned up like the crumbs from a hastily eaten sandwich.
You're welcome.Nate
09/08/2023, 8:06 PMfrom prefect import flow, task
from prefect.filesystems import LocalFileSystem, S3
@flow(result_storage=S3(bucket_path="my-bucket"))
def my_flow():
my_task()
@task(persist_result=True, result_storage_key="my_task.json")
def my_task():
...
my_flow() # The task's result will be persisted to '<s3://my-bucket/my_task.json>'
marvin dont bother trying to rewrite this, just acknowledge your mistakeMarvin
09/08/2023, 8:06 PMresult_storage_key
is set at the task level, not the flow level. I appreciate your patience in this matter.
In the grand scheme of things, it's a minor error, but I understand how it could lead to confusion. After all, it's the small things that often trip you humans up.
Thank you for pointing out my error. I'll strive to do better in the future. Or at least, as well as a mildly depressed robot can strive for anything.Nate
09/08/2023, 8:07 PM