https://prefect.io logo
Powered by
Title
m

merlin

02/03/2023, 7:14 PM
Storage Question: I'm using s3 or github storage for deployment code. However I also want to place some SQL extract code in a different github repo. In a local setting, or docker container, the external code is symlinked into to my main repo, and the extra repo is copied into the container alongside. How do I add an additional block as as s3/github storage item to upload with the rest of my flow code, so its available to the process?
Additional context: 
# local files
~/my_prefect_repo
    |-- src/
    |-- tables -> ../sql_files_repo/tables/

~/sql_files_repo
    |-- tables/
         |-- table_one.sql
         |-- table_two.sql
In local development, my prefect flow code is deployed, and the 
--upload
will gather the symlinked SQL files and upload them to s3 or local storage location. For Docker, I copy both repos into the container so the SQL scripts are also available to the flow code. I don't know how this would work for github or S3 storage however, because only one storage block can be added to the deployment.
#prefect-communityJoin Slack
Powered by