https://prefect.io
Join Slack
Hey, I've encountered an issue or misunderstanding with the prefect 2.0.0 manifests or deployment pa...
# ask-community
a
Hey, I've encountered an issue or misunderstanding with the prefect 2.0.0 manifests or deployment packages: I've configured a a GCS storage block called gcs-prefect-stprage and used this in the deployment build step. My folder structure is as follows:
Copy code
test
  |___ deployment.ymal
  |___ kayak_database_flows.py
My build command is (run from inside the folder "test")
Copy code
prefect deployment build kayak_database_flows.py:integrate_kayak_details --name integrate_kayak_db_details -t rmesbtest1 -sb gcs/gcs-prefect-stprage
Then i apply the deployment and run the deployment. Everything fine so far, the flow get's picked up by the corresponding agent However, the agent downloads ALL of the content of the GCS bucket. So really all of the content in this bucket, not only the stuff related to this deployment. Is this expected? If so, do I need a separate GCS storage block for any of the deployments?
👀 1
✅ 1
Maybe as an addon? How does one structure it's flow and deployment files and folders, if they have let's say 100 flows and 300 deployments?
a
this is expected but easily configurable
the path that you configure on your GCS block is configurable - add some directory to upload things there and also if you create a deployment from a project directory that contains only files relevant to your Prefect flow, Prefect will upload only those during build
let me try to build an example quickly to make it less ambiguous
a
Ah ok I see. If I theoretically put each deployment in a different folder, than during flow run, the agent only pulls the files in this folder? Cool, thanks. This works.
a
yes, Prefect pulls the directory of your project - this directory may be the same for multiple flows
LMK if the example helps or not
a
Yes this helps. Thanks! 🙌
🙌 1
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by