Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hello, does AWS S3 block storage work with S3-compatible service providers (such as Digital Ocean - Spaces)?

I setup the block with access-key and secret-key from Digital Ocean API , but encountered the error:

PermissionError: The AWS Access Key Id you provided does not exist in our records.
An exception occurred.

As the underlying library to setup this block uses boto3, does that mean that boto3 only checks auth against AWS S3 only (and not alternative storage service providers)?

Hi <@U01AYG8QZ4Y> - did you set the endpoint url in the AWSClientParameters to your digital ocean endpoint url?

I tried this out and was able to upload a file using the S3 Bucket block with an AwsCredentials instance that had my spaces endpoint url in the client params

Hello <@U02GMEZU18B>!

I have not tried to upload files using Python in my flow code, as my objective is to use DigitalOcean spaces as storage for my deployments.

I tried the following (using the Prefect Cloud UI):
1. Create S3 Bucket Block 
    a. Specify bucket name
2. Create AWS Credentials Block
    a. Enter key ID
    b. Enter key secret
    c. Enter DigitalOceans endpoint URL
The S3 bucket block is now created and linked with AWS Credentials, with AWSClientParameters populated as per step 2 above.

Now, when I try to build a deployment that specifies this storage bucket, I run into the below error:

Build command (shortened for brevity)
```prefect deployment build /path/to/flows \
-sb do-prefect-orion```
Error Message
```Found flow 'xxxxxxxx'
Traceback (most recent call last):
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 230, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 181, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/root/orion/orion-venv/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/root/.pyenv/versions/3.8.9/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/root/.pyenv/versions/3.8.9/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/root/orion/orion-venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/cli/deployment.py", line 936, in build
    block_type, block_name, *block_path = storage_block.split("/")
ValueError: not enough values to unpack (expected at least 2, got 1)
An exception occurred.```

hi!

that error looks to be caused by the fact you’re passing `do-prefect-orion` instead of `s3/do-prefect-orion` , when passing infra or storage blocks to the `deployment build` command you have to pass `block-type/block-instance-name`

That worked perfectly, thank you !

(Verified the block slug name with `prefect block ls` )