Adam
07/07/2022, 7:46 PMKyle McChesney
07/07/2022, 7:49 PMmap
, which will allow you to have 1 task per csv, and then capture the idea that some worked and some failed. Prefect should be able to recover and re-run just the failed mapped task instances.Andrew Huang
07/07/2022, 7:56 PMimport pandas as pd
from prefect import flow, task
@task(retries=2, retry_delay_seconds=5)
def download_csv(url):
return pd.read_csv(url)
@flow
def download_flow(urls):
dfs = [download_csv(url) for url in urls]
return dfs
Adam
07/07/2022, 8:05 PMAndrew Huang
07/07/2022, 8:06 PM