https://prefect.io
Join Slack
*Storage Question*: I'm using s3 or github storage for deployment code. However I also want to plac...
# ask-community
m
Storage Question: I'm using s3 or github storage for deployment code. However I also want to place some SQL extract code in a different github repo. In a local setting, or docker container, the external code is symlinked into to my main repo, and the extra repo is copied into the container alongside. How do I add an additional block as as s3/github storage item to upload with the rest of my flow code, so its available to the process?
Additional context:
Copy code
# local files
~/my_prefect_repo
    |-- src/
    |-- tables -> ../sql_files_repo/tables/

~/sql_files_repo
    |-- tables/
         |-- table_one.sql
         |-- table_two.sql
In local development, my prefect flow code is deployed, and the 
--upload
will gather the symlinked SQL files and upload them to s3 or local storage location. For Docker, I copy both repos into the container so the SQL scripts are also available to the flow code. I don't know how this would work for github or S3 storage however, because only one storage block can be added to the deployment.
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by