import subprocess
from prefect import flow, task
def run_scripts():
spiders = ['fishykart', 'bunnycart', 'finsnflora','aquabynature','aquaplant','aquaticplants','geturpet','sreepadma']
for spider in spiders:
subprocess.run(['pipenv', 'run', 'scrapy', 'crawl', spider])
run_scripts()
how to i add @task and @flow can someone give me the code so i can deploy to run it daily at 9am
h
Hen rik
11/10/2023, 9:48 PM
Hi, I'm also new to prefect. But I did it like this. Hope it helps
Copy code
import subprocess
from prefect import flow, task
@flow()
def run_scripts():
spiders = ['fishykart', 'bunnycart', 'finsnflora', 'aquabynature', 'aquaplant', 'aquaticplants', 'geturpet',
'sreepadma']
for spider in spiders:
subprocess.run(['pipenv', 'run', 'scrapy', 'crawl', spider])
if __name__ == "__main__":
run_scripts.serve(name="run_scripts", cron="0 9 * * *")
m
Manal Naik
11/12/2023, 2:26 PM
hello thanks for the code i have deployed it successfully and tried a quick run but the status is late
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.