https://prefect.io logo
Title
n

Nicholas Torba

04/28/2023, 6:29 PM
Moving this post here since this seems more relevant: Has anyone been struggling with getting the agent to now stall on downloading code before running a deployment? I'm downloading a good bit of code, but when I run a manual s3 download on command line, it completes quickly. Also, I can run this code snippet:
from prefect.filesystems import S3
s3_block = S3.load("flow-code")
s3_block.get_directory(from_path="/", local_path=".")
and it also downloads in a couple of seconds however, when I run the prefect-agent, it stalls at the download. It stalls whether I use an S3 storage block or if I try to run it locally. Frustrating because everything was working fine until it just... wasn't anymore. Has anyone else seen this? One difference I see in the code snippet to what is happening in prefect-agent code is that it is being run async, but I didn't think that should be an issue
c

Chris White

04/28/2023, 8:16 PM
Hey Nicholas - I've seen this happen before when the s3fs library and fsspec get out of sync / have incompatible versions, so I suggest checking what versions you have locally and what versions are running in your deployment's execution environment
n

Nicholas Torba

04/28/2023, 8:21 PM
hmmm, I'm running the agent on my laptop right now. I can also run this code locally and it works fine
from prefect.filesystems import S3
s3_block = S3.load("amplify-flow-code")
s3_block.get_directory(from_path="", local_path=".")
c

Chris White

04/28/2023, 8:24 PM
and you're not using virtual environments or docker or anything like that?
n

Nicholas Torba

04/28/2023, 8:24 PM
I’m running the agent in the same conda env im running that code snippet in
Interestingly, it also hangs there when I try to run it with local code
c

Chris White

04/28/2023, 8:31 PM
innnnteresting, the plot thickens
i think you should open an issue for this on github
n

Nicholas Torba

04/28/2023, 8:33 PM
doing that now. will send link in a minute
:thank-you: 1
so I made a simple flow example, and it works fine. In my more complex one, I am trying to include a django application in my code so I can access the django models and db from the task. Something about copying over the entire app is causing it to freeze? Trying to figure out how to reproduce it for the ticket
r

Robin

05/01/2023, 4:47 PM
I have similar issues with big flow code. In my case, I have an EC2 instance on which I have the code, I deploy it without specifying a storage (hence using local storage?) and then I run the agent on that EC2 instance. However, download also takes a long time (although it is local, so what takes long time? does it copy code over locally?). Unfortunately, there are even errors
No such file or directory
when downloading the code, so that the deployment currently does not run successfully 😕
Could it be related to
micromamba
vs
conda
?