Hi , is there a way to deploy multiple data pipeli...
# prefect-community
Hi , is there a way to deploy multiple data pipelines(one for daily and other pipeline for weekly etc) , would appreciate the help , thanks in advance
Tons of ways
You create a deployment for each of the pipelines, and then you set the schedule that you want for each of them.
Thank you so much for your help , so basically i woul get multiple yaml files for multiple data pipelines and i just need to deploy them , so let us say i create 3 data pipelines and once i deploy them they would give me 3 sh files , so i just need to run the 3 sh files and everything should work as expected or my understanding was wrong? thanks for the quick responces
We don’t create any sh files
How familiar with Prefect are you?
On mobile but I’ll try to answer - You create your flows with the prefect @flow and @task decorators on your python functions in your data pipeline script. You then create a deployment, which will produce a YAML file. When you create the deployment you can specify a schedule, or you can add a schedule to a deployment later using the UI. You the apply the deployment with the CLI command ‘prefect deployment apply <path to deployment yaml>’ Then they will run as long as you have an agent running
If you’re not very familiar with prefect, you should try the tutorial for prefect 2 on our docs page. It will get you up to speed quickly
Then I should be able to help you better when you bump into concepts that are unclear
Gtg for an hour or so, but I’ll check back on this
I apologize i was following an medium article and when the deployment build --apply was applied they showed the output in the shell script and i assumed that prefect would create one my bad , i should have read the article better but thanks for all the help , will get back to you if i have any other queries!
Sounds great and good luck!