hello i have mined data using scrapy and integrated it with dlt pipeline to push on duckdb .. i have scraped multiple sides so i have multiple files ..need help with creating a prefect cron file so i can add all the commands to run all of my scrapy files so data gets pushed everyday
t
Tess Dicker
10/31/2023, 3:45 PM
hey @Manal Naik -- You can follow the Getting Started documentation to begin your workflow. Then you can schedule that file to run in a few ways, shown in this Schedules Documentation!
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.