https://prefect.io logo
Title
s

Samuel Hinton

03/06/2023, 10:31 AM
Hi all! Yesterday I had all my flows running properly, but all my flows today have had failed with the same error. I can no longer even get a simple toy model working, every flow responds with
No such file or directory
If anyone has any ideas as to what might be causing issues with S3 block file downloading or a way to debug it, please let me know!
👀 1
r

redsquare

03/06/2023, 10:36 AM
I have the same issues all weekend but been too ill to debug it
s

Samuel Hinton

03/06/2023, 10:37 AM
Wow, Ive got like 10 prefect github issues open, but didnt see that one. Good catch mate, I’ll roll the version back!
Literally a lifesaver
r

redsquare

03/06/2023, 10:38 AM
yeah we have just wasted 3 hours looking at it this morning
😅 1
s

Samuel Hinton

03/06/2023, 10:41 AM
Adding
path="/"
to the deploy fixes things as well
r

redsquare

03/06/2023, 10:42 AM
ok cool, watch that fix break again at a later date
I am loathed to start pinning lots of libs as your forever chasing tails updating
s

Samuel Hinton

03/06/2023, 10:48 AM
Yeah I just spent a week unpinning things because of so many issues we had locally (looking at apple here, because supporting amd64 and M1 chips on things like tensorflow and ray is such a massive PITA). Really trying to keep versions loose until I have to bite the bullet
r

redsquare

03/06/2023, 10:50 AM
naughty I guess of s3fs to change this on a point release
I need to audit the requirements so I can easily compare between runs I guess
s

Samuel Hinton

03/06/2023, 10:51 AM
Ya
d

Dave Chater

03/06/2023, 11:03 AM
Cheers @redsquare
😁 1
👍 1
👀 1
r

redsquare

03/06/2023, 11:04 AM
thinking out loud - maybe a github action on deploy that writes the deployed deps & versions to a diff repo, to allow for easy audit/diagnosing