Hi guys, can I use S3 block to download files from...
# prefect-communitya
Anat Tal Gagnon
08/09/2022, 9:13 AMHi guys, can I use S3 block to download files from S3 bucket?
or am I just using the block as secret to the AWS S3Download task? any examples?
a
Anna Geller
08/09/2022, 9:42 AM
Yes, totally! both works, up to you which one you prefer
a
Anat Tal Gagnon
08/09/2022, 9:43 AMThanks Anna! You got a code snip as example?
a
Anna Geller
08/09/2022, 9:53 AM
Copy code
from prefect.filesystems import S3
s3_block = S3.load("dev")
s3_block.put_directory("local_path", "s3_path")Anna Geller
08/09/2022, 9:53 AM
just an example
Anna Geller
08/09/2022, 9:54 AM
you can inspect the block methods in code to see more how this works, docstrings are quite good
🙏 1
a
Anat Tal Gagnon
08/09/2022, 2:01 PM@Anna Geller I'm getting an error "*RuntimeWarning: coroutine 'S3.put_directory' was never awaited*"
Anat Tal Gagnon
08/09/2022, 2:04 PMUntitled.py
Anat Tal Gagnon
08/09/2022, 2:13 PMa
Anna Geller
08/09/2022, 5:21 PM
You'd likely need await in front of it
e
eddy davies
08/17/2022, 11:02 AMIs generating a local file and then pushing it to s3 as shown here the best practice? Rather than saving a file straight to s3?
eddy davies
08/17/2022, 11:49 AMI tried to create a super basic attempt at using await but it gives a massive error.
a
Anna Geller
08/17/2022, 1:02 PM
What are you trying to do Eddy?
e
eddy davies
08/18/2022, 12:17 PMSave data to S3, though I do notice now that the start of this thread was downloading a file instead...
a
Anna Geller
08/18/2022, 1:32 PM
this works for me, need to check how to use it with Blocks:
Copy code
import fsspec
fs = fsspec.filesystem("s3")
fs.put("/Users/anna/repos/dataflow-ops/docs", "<s3://prefect-orion/docs>", recursive=True)e
eddy davies
08/18/2022, 1:41 PMGreat, that works! I am not providing any AWS Access Key, is it using that from my block somehow then?
a
Anna Geller
08/18/2022, 1:44 PM
from your ~/.aws/credentials
👍🏻 1
e
eddy davies
08/19/2022, 1:55 PMhow can I guarantee this will run when deployed too production, say using dask, docker and/or kubernetes?
319 Views