@Marvin can i set result_storage to a s3 block if I want to persist data of a flow?

Yes, you can set result storage to an S3 block in Prefect 2.0 if you want to persist the data of a flow. You can configure a new storage block based on the S3 block type through the Prefect UI or by using the Prefect Python API. Here is an example of how to create an S3 block using the Python API:

python
from prefect.filesystems import S3

block = S3(bucket_path="my-bucket/a-sub-directory",
           aws_access_key_id="foo", 
           aws_secret_access_key="bar")

block.save("example-block")

After creating the block, it will be available for use by anyone with appropriate access to your Prefect API. You can use this block to build a deployment by passing its slug to the

prefect deployment build

command. The storage block slug is formatted as

block-type/block-name

. For more detailed information, you can review the storage documentation. Prefect 2.0 simplifies many usage patterns and provides a seamless onboarding experience. Every time you run a flow, whether it is tracked by the API server or run ad-hoc through a Python script, it appears on the same UI page for easier debugging and observability. Remember that Prefect 2.0 treats your functions as your flows and tasks, with automatic detection of flows and tasks without needing a rigid DAG structure. Using tasks is encouraged to provide maximum visibility into your workflows.

@Marvin so this will work? from prefect.filesystems import S3 s3_block = S3.load("bucket-name") @flow(log_prints=True, persist_result=True, result_storage = s3_block ) def hello()

The

S3.load("bucket-name")

method is used to load an S3 block from storage with the given name, assuming that you've already saved this block with the name

"bucket-name"

in Prefect Cloud or Prefect Server. Your

@flow

decorator setup looks generally correct, with

persist_result=True

ensuring that the flow's result is persisted, and

result_storage=s3_block

indicating that the S3 block should be used as the result storage. However, you may need to ensure that you've set up the S3 block with all required credentials and configurations, and that you've saved it properly. If any of these conditions are not met, you may encounter issues when trying to use the S3 block for result storage. Here is the tweaked code for clarity:

python
from prefect import flow
from prefect.filesystems import S3

# Load an existing S3 block from storage
s3_block = S3.load("bucket-name")

@flow(log_prints=True, persist_result=True, result_storage=s3_block)
def hello():
    print("Hello, world!")

# Running the flow
hello()

Make sure to replace

"bucket-name"

with the actual saved name of your S3 block. If you haven't saved the S3 block yet, you'll need to configure and save it first using, for example:

python
from prefect.filesystems import S3

s3_block = S3(bucket_path="my-bucket/path", aws_access_key_id="your-key", aws_secret_access_key="your-secret")
s3_block.save("bucket-name")

After saving the block with the name

"bucket-name"

, you can then use

S3.load("bucket-name")

to load it for use with the

@flow

decorator as demonstrated earlier.