https://prefect.io
Join Slack
Hi guys, can I use S3 block to download files from S3 bucket? or am I just using the block as secret...
# ask-community
a
Hi guys, can I use S3 block to download files from S3 bucket? or am I just using the block as secret to the AWS S3Download task? any examples?
a
Yes, totally! both works, up to you which one you prefer
a
Thanks Anna! You got a code snip as example?
a
Copy code
from prefect.filesystems import S3

s3_block = S3.load("dev")
s3_block.put_directory("local_path", "s3_path")
just an example
you can inspect the block methods in code to see more how this works, docstrings are quite good
🙏 1
a
@Anna Geller I'm getting an error "*RuntimeWarning: coroutine 'S3.put_directory' was never awaited*"
Untitled.py
Untitled.py
a
You'd likely need await in front of it
e
Is generating a local file and then pushing it to s3 as shown here the best practice? Rather than saving a file straight to s3?
I tried to create a super basic attempt at using await but it gives a massive error.
s3_test.py
a
What are you trying to do Eddy?
e
Save data to S3, though I do notice now that the start of this thread was downloading a file instead...
a
this works for me, need to check how to use it with Blocks:
Copy code
import fsspec

fs = fsspec.filesystem("s3")
fs.put("/Users/anna/repos/dataflow-ops/docs", "<s3://prefect-orion/docs>", recursive=True)
e
Great, that works! I am not providing any AWS Access Key, is it using that from my block somehow then?
a
from your ~/.aws/credentials
👍🏻 1
e
how can I guarantee this will run when deployed too production, say using dask, docker and/or kubernetes?
375 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by