https://prefect.io
Join Slack
Did something change with how deployed flows run with the `path` argument specified? I’m using S3 st...
# ask-community
e
Did something change with how deployed flows run with the 
path
argument specified? I’m using S3 storage and for each deployment I specify a custom `path`:
Copy code
Deployment.build_from_flow(flow=flow, infrastructure=ECSTask.load('...'), storage=S3.load('...'), name=my_custom_name, path=my_custom_name, apply=True)
my flow assumes that the files in storage are pulled to the root during flow runs but _now it seems like it’s pulled as a subdirectory of 
my__custom_name
z
Which version?
e
I’m on prefect v2 but let me get all that info
• Using Prefect Cloud • Base image from 
prefecthq/prefect:2-python3.10
-> 
prefecthq/prefect:2-python3.11
• prefect-aws : 
0.2.5
• prefect: 
2.8.4
• s3fs: 
2023.3.0
z
Oops 🙂
Oh weird GH didn’t change my URL when I navigated
🙂
e
No the flow storage is being pulled. I added a
Copy code
shell_run_command("ls -la")
to my task and seeing that the remote storage (with 
path
specified on the deployment) is in a subdirectory
@Zanie it does look like I upgraded s3fs today which is when the error started
z
I’d be quite suspicious of that
It’s not clear what the issue is though
If you’re using it outside a deployment that might be really helpful for establishing a minimal example
e
Using what outside a deployment? I’ll try to create a small example, but that strikes me as a lot of moving parts
z
Ah I see sorry I misunderstood
It is indeed a lot of moving parts
Perhaps showing where things are downloaded with and without the new version would help?
e
OK I can confirm that 
s3fs
is the issue. I downgraded and reran the flow and it’s working
I’ll try to write up an issue
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by