@Marvin I updated Prefect to the latest version and now I'm getting a FileNotFoundError when running an existing flow.

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 293, in aresult
    return await asyncio.wrap_future(self.future)
  File "/usr/local/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 318, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments/steps/pull.py", line 28, in set_working_directory
    os.chdir(directory)
FileNotFoundError: [Errno 2] No such file or directory: '/home/prefect/hq_integration'

The directory does exist and was not changed. Ideas?

It seems you're encountering a

FileNotFoundError

after updating Prefect, which implies there may be some compatibility issues or bugs introduced with the new version. Here are two potentially relevant GitHub issues you might find helpful: 1. One user reported a similar

FileNotFoundError

while trying to run Prefect and tracked it down to changes in how Prefect sets the working directory when creating the deployment. You can check out this issue for more information. 2. Another issue reported problems with doubled paths when using

LocalFileSystem

storage blocks, which may be related to your issue since it also involves file paths not being found as expected. You might want to look at GitHub issue 11295 for further details. Unfortunately, neither document searches nor the issue searches provided a direct solution to the problem. Therefore, it could be helpful to check the GitHub issues I mentioned above to see if they provide any insights into your problem. If they don't help, consider opening a new issue on the Prefect GitHub repository to get specific assistance with your case.

For anyone else reading this, I broke it in a roundabout way. First, I logged into the server running the agent (EC2) and upgraded Prefect to the latest version. Then I updated Ubuntu with apt upgrade and pruned all the leftover Docker containers that Prefect and/or Docker leaves lying around with

prune

. I totally uninstalled and reinstalled Docker, and that's when I noticed the Docker service wouldn't start and was throwing an error about my logging options being invalid. I manually pulled the Prefect image I use with Docker and then it worked again. My guess is that it was using a cached container that I pruned, and it wouldn't pull the image and make a new one. Or something, I dunno, I'm new to Docker. 🙃

glad you figured it out!