https://prefect.io logo
Title
c

CA Lee

02/18/2023, 4:26 AM
Hello, does AWS S3 block storage work with S3-compatible service providers (such as Digital Ocean - Spaces)?
I setup the block with access-key and secret-key from Digital Ocean API , but encountered the error: PermissionError: The AWS Access Key Id you provided does not exist in our records. An exception occurred. As the underlying library to setup this block uses boto3, does that mean that boto3 only checks auth against AWS S3 only (and not alternative storage service providers)?
n

Nate

02/20/2023, 7:40 PM
Hi @CA Lee - did you set the endpoint url in the AWSClientParameters to your digital ocean endpoint url? I tried this out and was able to upload a file using the S3 Bucket block with an AwsCredentials instance that had my spaces endpoint url in the client params
c

CA Lee

02/27/2023, 3:24 AM
Hello @Nate! I have not tried to upload files using Python in my flow code, as my objective is to use DigitalOcean spaces as storage for my deployments. I tried the following (using the Prefect Cloud UI): 1. Create S3 Bucket Block a. Specify bucket name 2. Create AWS Credentials Block a. Enter key ID b. Enter key secret c. Enter DigitalOceans endpoint URL The S3 bucket block is now created and linked with AWS Credentials, with AWSClientParameters populated as per step 2 above. Now, when I try to build a deployment that specifies this storage bucket, I run into the below error: Build command (shortened for brevity)
prefect deployment build /path/to/flows \
-sb do-prefect-orion
Error Message
Found flow 'xxxxxxxx'
Traceback (most recent call last):
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 230, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 181, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/root/orion/orion-venv/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/root/.pyenv/versions/3.8.9/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/root/.pyenv/versions/3.8.9/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/root/orion/orion-venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/root/orion/orion-venv/lib/python3.8/site-packages/prefect/cli/deployment.py", line 936, in build
    block_type, block_name, *block_path = storage_block.split("/")
ValueError: not enough values to unpack (expected at least 2, got 1)
An exception occurred.
n

Nate

02/27/2023, 3:33 AM
hi! that error looks to be caused by the fact you’re passing
do-prefect-orion
instead of
s3/do-prefect-orion
, when passing infra or storage blocks to the
deployment build
command you have to pass
block-type/block-instance-name
1
c

CA Lee

02/27/2023, 6:19 AM
That worked perfectly, thank you ! (Verified the block slug name with
prefect block ls
)