https://prefect.io logo
Title
m

Mike Safruk

04/19/2023, 9:46 PM
Hey Prefect Community, I'm having a issue with running a deployment using a docker container. When I run the flow I deployed into a S3 bucket (other deployments using this bucket are working), I get the following error:
Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 879, in exec_module File "<frozen importlib._bootstrap_external>", line 1016, in get_code File "<frozen importlib._bootstrap_external>", line 1073, in get_data FileNotFoundError: [Errno 2] No such file or directory: '/opt/flow-code.py' The above exception was the direct cause of the following exception: ScriptError: Script at 'flow-code.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
Has anyone seen this before? Any tips on where to spend time troubleshooting?
✅ 1
a

Abhinav Chordia

04/19/2023, 9:52 PM
How are you deploying?
m

Mike Safruk

04/19/2023, 9:53 PM
Via command line using prefect deployment build command
c

Chris White

04/19/2023, 9:55 PM
Hey Mike! It's hard to know exactly but we have been experimenting with some more explicit and transparent mechanics for deployments in the form of "projects"; I've seen good success with users who transition to the project structure get away from these issues: https://docs.prefect.io/latest/tutorials/projects/ In your case, you're probably going to want to use the
docker-s3
recipe (
prefect project init --recipe docker-s3
)
m

Mike Safruk

04/19/2023, 9:57 PM
Hmmm how hard is it to use the new Project way? I have some deadlines and I don't have much time to experiment with something new
a

Abhinav Chordia

04/19/2023, 9:57 PM
I’ve seen that issue above when there was a conflict due to the version of the fsspec library and the remote storage library you are using (s3fs)
👀 1
If you pin it to the same version as your s3fs lib, it might go away.
c

Chris White

04/19/2023, 9:59 PM
it shouldn't be hard, the biggest change would probably be that you'll need to use a worker instead of an agent (but assuming you have a standard setup it shouldn't be more than a new CLI command to start things). I'd say try Abhinav's suggestion (they're knowledgeable here so probably are on the right track!), and if you want to spend ~15 minutes with the new project structure that should be sufficient to get things working, but obviously no pressure if you solve it in other ways
m

Mike Safruk

04/19/2023, 9:59 PM
OK thanks Chris and Abhinav. I'll try the version path first! Appreciate the guidance
👍 1
Using older s3fs and fsspec versions did the trick! Thanks again
🙌 1
j

Jean-Michel Provencher

05/03/2023, 6:57 PM
Hello, it's kind of weird to have to downgrade s3fs and fssspect versions in order to not have an error when flows are running. I understand the project proposition, but what you did is a pretty major breaking changes and it's preventing us to upgrade without completely changing the way we deploy flow. Could you point us to what is actually causing the script not being able to pickup the file properly from s3?
FileNotFoundError: [Errno 2] No such file or directory: 'prefect2/test.py'
file is actually in S3, so it's probably a change in the way you handle paths with the S3 storage right ?
(I'm using the Deployment.build_from_flow python method to deploy)
I assume something has changed in the internal mecanism to retrieve the file when running a flow because all the files are properly uploaded to the S3 storage.
c

Chris White

05/03/2023, 7:12 PM
Hi Jean - projects have not affected or altered how previous deployments work so this isn't projects related; the s3fs / fsspec issue is an issue with those libraries which are libraries Prefect uses to upload / download code from S3
j

Jean-Michel Provencher

05/03/2023, 7:14 PM
I tried updating s3fs to the same version as fsspec and it was not working. I'll try digging more. Any documentation on what s3fs version should be use with the latest prefect version ?
bundling it inside the prefect pip package would probably avoid having this issue too maybe.
(but there's other implications 😄 )
s3fs==2023.4.0
fsspec==2023.4.0
I have these two fixed versions, but still prefect is no longer able to pull the files from the S3 storage
c

Chris White

05/03/2023, 7:29 PM
I haven't experienced this issue myself so am not sure what versions cause it - two things to confirm: • have you confirmed that these package versions are also the ones present in your deployment's runtime environment? • what block type are you using? you'll want to make sure to use
prefect.filesystems.S3
j

Jean-Michel Provencher

05/03/2023, 7:31 PM
prefect 2.10.6 depends on fsspec!=2023.3.0 and >=2022.5.0
I'm using prefect.filesystems.S3 which requires to manually install s3fs. However, in the latest prefect version, fsspec has been upgraded, and it does not seem to work with s3fs anymore
so basically prefect does not work with any version of s3fs greater than 0.6.0?
I find the root cause. I was able to fix by adding a trailing "/" to the "path" attribute when creating a flow. IMO this should probably be handled in a new fix to make sure it does not break existing integrations.
path=deployment_variables.short_commit_hash + "/", # Fix in order for prefect to be able to properly fetch the storage code from S3  <https://github.com/PrefectHQ/prefect/issues/9262>
c

Chris White

05/04/2023, 4:04 PM
Apologies for not seeing this yesterday - that's interesting and seems like a potentially simple fix for a number of open issues; thank you for following through on investigating this! I'll review those issues and try to confirm this wouldn't have any adverse behavior
👀 1
@Jean-Michel Provencher opened a PR for this that we'll try to get into today's release! https://github.com/PrefectHQ/prefect/pull/9440
j

Jean-Michel Provencher

05/04/2023, 5:11 PM
thanks!