Hi! I am using the prefect REST API to create prefect_aws blocks. I successfully created a aws-credentials block, but when I try to create a s3-bucket block (in prefect cloud, here it is the official doc), I don't know how to relate the previous block for credentials. I tried to relate the block manually in the web UI. Then, if I get the block using the API for the s3-bucket block I see the plain credentials (aws_access_key_id, aws_secret_access_key, ...). How can I reference the aws-credentials block to the s3-bucket block? Thank you!

hi @jcozar - happy to help if you can share what you've tried / what part (in terms of your http client or whatever you're using) you're stuck at

Hi @Nate, of course! I created a aws-credential block and s3-bucket block like this:

base_url = f"<https://api.prefect.cloud/api/accounts/{prefect_secrets['PREFECT_ACCOUNT_ID']}/workspaces/{prefect_secrets['PREFECT_WORKSPACE_ID']}>"

    # Authorization headers
    headers = {
        "Authorization": f"Bearer {prefect_secrets['PREFECT_API_KEY']}",
        "Content-Type": "application/json"
    }

# Create aws credentials block
slug = "aws-credentials"
# created on 2022-11-16T17:43:57.765962Z
block_type_id = "2b9baa6c-86df-4019-9711-392fb5b937d9"
    
# created on 2025-01-16T17:28:06.240831Z
block_schema_id = "6674cc7d-3545-4cdf-9a12-eab2199f22c2"
    
# build data
credentials_name = "mycredentials"
access_key_id = "..."
secret_access_key = "..."
name = "mycredentials"
data = {
    "aws_access_key_id": access_key_id,
    "aws_secret_access_key": secret_access_key,
    "region_name": "eu-west1",
}

url = f"{base_url}/block_documents"
response = requests.post(url, headers=headers, json={
    "name": name,
    "block_type_id": block_type_id,
    "block_schema_id": block_schema_id,
    "data": data
})
block_credentials_id = response.json()["id"]

# Create s3-bucket block
slug = "s3-bucket"
# created on 2023-02-19T09:16:43.906341Z
block_type_id = "e7ef6d0d-d49b-4e8d-970c-a2e66b35a872"

# created on 2025-01-16T17:38:14.090658Z
block_schema_id = "8e52f2cb-4385-4cef-9652-c3bf16bf1f5a"

# build data
name = "mybucket"
data = {
    "bucket_name": "mybucket",
    "credentials": block_credentials_id
}
url = f"{base_url}/block_documents"
response = requests.post(url, headers=headers, json={
    "name": name,
    "block_type_id": block_type_id,
    "block_schema_id": block_schema_id,
    "data": data
})

It works correctly, except that the block mybucket is created with none credentials, like it is shown in the snapshot

if you're using python, is there a reason you don't want to use

prefect-aws

?

» uv run --with prefect-aws ipython
#[1]
from prefect_aws import AwsCredentials, S3Bucket

#[2]
AwsCredentials().save('test') # uses default creds a la boto3
Out[2]: UUID('053ddae0-69cc-4960-bc55-41001c75596c')

#[3]
S3Bucket(bucket_name='xyz', credentials=AwsCredentials.load('test')).save('test', overwrite=True)
Out[3]: UUID('7651bbde-52d7-474f-80b1-561688bb057f')

#[4]
S3Bucket.load('test').list_objects()
22:58:16.233 | INFO    | prefect.S3Bucket - Listing objects in bucket
Out[4]:
[{'Key': '21ee2d65bbc14f699d8823d345feea37',
  'LastModified': datetime.datetime(2024, 10, 30, 19, 37, 1, tzinfo=tzutc()),
  'ETag': '"92a28d3cfff4b8a201e3e818d34c3b2c"',
  'Size': 669,
  'StorageClass': 'STANDARD'},
 {'Key': '72b4f5acf83045399f5504d7b4806b67',
  'LastModified': datetime.datetime(2024, 10, 30, 19, 37, tzinfo=tzutc()),
  'ETag': '"92a28d3cfff4b8a201e3e818d34c3b2c"',
  'Size': 669,
  'StorageClass': 'STANDARD'}]

otherwise you can do it (with significantly more code) via REST API like this

Perfect, it works like charm! Thank you @Nate. Please, can you post a link to the official doc for $ref usage? It would be really helpful in my next steps

better described in the API REST doc than in the python SDK

well, we have the SDK to make it easier to do these things 🙂 so hopefully it is indeed a bit easier here is the documentation on blocks, we don't have specific documentation for using blocks against the REST API because we typically would recommend the SDK or client methods to reiterate, that 100 line script against the REST api is a long way of doing the following

» uv run --with prefect-aws ipython
#[1]
from prefect_aws import AwsCredentials, S3Bucket

#[2]
AwsCredentials().save('test') # uses default creds a la boto3
Out[2]: UUID('053ddae0-69cc-4960-bc55-41001c75596c')

#[3]
S3Bucket(bucket_name='xyz', credentials=AwsCredentials.load('test')).save('test', overwrite=True)
Out[3]: UUID('7651bbde-52d7-474f-80b1-561688bb057f')

create things like a workpool

CLI

» prefect work-pool create local --type process
Created work pool 'local'!

client methods

#[1]
from prefect import get_client

#[2]
from prefect.client.schemas.actions import WorkPoolCreate

#[3]
async with get_client() as client:
    print(await client.create_work_pool(WorkPoolCreate(name='testing', type='docker')))

id=UUID('bf99a0fd-864f-4420-8388-863c383cc2fd') name='testing' description=None type='docker' base_job_template={} is_paused=False concurrency_limit=None status=WorkPoolStatus.NOT_READY default_queue_id=UUID('35e00b81-cec1-4fde-914e-02c65496362d')

and similarly we have conveniences for different types of concurrency limits

Ok thank you for the feedback and for the links! 🙂