Hi , is there a way to deploy multiple data pipeli...
# ask-communityr
Revanth
11/09/2022, 2:33 PMHi , is there a way to deploy multiple data pipelines(one for daily and other pipeline for weekly etc) , would appreciate the help , thanks in advance
✅ 1
p
Peyton Runyan
11/09/2022, 3:03 PMTons of ways
Peyton Runyan
11/09/2022, 3:03 PMYou create a deployment for each of the pipelines, and then you set the schedule that you want for each of them.
Peyton Runyan
11/09/2022, 3:04 PMPeyton Runyan
11/09/2022, 3:04 PMPeyton Runyan
11/09/2022, 3:05 PMr
Revanth
11/09/2022, 3:36 PMThank you so much for your help , so basically i woul get multiple yaml files for multiple data pipelines and i just need to deploy them , so let us say i create 3 data pipelines and once i deploy them they would give me 3 sh files , so i just need to run the 3 sh files and everything should work as expected or my understanding was wrong? thanks for the quick responces
p
Peyton Runyan
11/09/2022, 3:53 PMWe don’t create any sh files
Peyton Runyan
11/09/2022, 3:54 PMHow familiar with Prefect are you?
Peyton Runyan
11/09/2022, 3:57 PMOn mobile but I’ll try to answer -
You create your flows with the prefect @flow and @task decorators on your python functions in your data pipeline script.
You then create a deployment, which will produce a YAML file. When you create the deployment you can specify a schedule, or you can add a schedule to a deployment later using the UI.
You the apply the deployment with the CLI command ‘prefect deployment apply <path to deployment yaml>’
Then they will run as long as you have an agent running
Peyton Runyan
11/09/2022, 3:58 PMIf you’re not very familiar with prefect, you should try the tutorial for prefect 2 on our docs page. It will get you up to speed quickly
Peyton Runyan
11/09/2022, 3:59 PMThen I should be able to help you better when you bump into concepts that are unclear
Peyton Runyan
11/09/2022, 3:59 PMGtg for an hour or so, but I’ll check back on this
r
Revanth
11/09/2022, 4:00 PMI apologize i was following an medium article and when the deployment build --apply was applied they showed the output in the shell script and i assumed that prefect would create one my bad , i should have read the article better but thanks for all the help , will get back to you if i have any other queries!
p
Peyton Runyan
11/09/2022, 4:56 PMSounds great and good luck!
177 Views