https://prefect.io logo
Title
m

Mitch Comardo - Lyric Research

04/28/2023, 8:40 PM
Hi All - Relatively new here. Trying to work through a simple example of deploying a prefect project that uses s3 storage. Is it common for this method to take 30-60 minutes on a 2 file python project?
c

Chris White

04/28/2023, 8:41 PM
Hey Mitch - absolutely not normal, sorry you're dealing with that! could you describe where you're running into issues?
m

Mitch Comardo - Lyric Research

04/28/2023, 8:44 PM
Sure - I have a entrypoint file that simply imports another python file that for now just contains a function with a
pass
statement in it. It also has the following deployment definition...
I've been stuck here for around 35 minutes now:
it looks like it's sending data over to the block based on the events firing:
c

Chris White

04/28/2023, 8:49 PM
hm interesting, i've seen some weird behavior when fsspec and s3fs versions are incompatible; could you try using the block directly to see if that's where the hang is? Something like:
s3_bucket_block.put_directory(__file__)
m

Mitch Comardo - Lyric Research

04/28/2023, 8:52 PM
looks like it still hangs there...
c

Chris White

04/28/2023, 8:58 PM
interesting! so in that case, i suggest runing
pip freeze
and looking specifically at the versions of
fsspec
and
s3fs
m

Mitch Comardo - Lyric Research

04/28/2023, 9:00 PM
image.png,image.png
@Chris White it actually appears to be the deployment build that it's getting hung up on. That process basically doesnt finish even from the cli command
prefect deployment build -n nightly-symbol-download -p lyric-aws -sb s3-bucket/lyric-s3 flows/nightly_symbol_download.py:nightly_symbol_download
And narrowing it down once more it appears to be the -sb command in the build process. When I omit that, the build happens instantly.
OK I solved this. The problem was using the Block
s3-bucket
wasn't doing anything... so I found the
s3
block and it appears to be working. Onto the next step!
c

Chris White

04/29/2023, 12:29 AM
Sorry I stepped away earlier - huh that’s interesting to hear the bucket block was a dud…