p

    Preston Marshall

    2 years ago
    Ok, another question. Would you mind describing at a high level how I would, on a schedule: reach out to an SFTP server, list the files (dynamic workflows!), copy them somewhere, and send a slack notification? I think the dynamic part has me the most confused right now, the rest is just stringing tasks together from the library
    j

    josh

    2 years ago
    Here’s how I would set it up: • First task which reaches out to the SFTP server, gets the information back and returns it • Downstream task which maps over the results from the upstream fetch and copies them somewhere https://docs.prefect.io/core/concepts/mapping.html#prefect-approach (this is where the dynamic task creation part happens) • Either use a slack state handler https://docs.prefect.io/core/tutorials/slack-notifications.html#slack-notifications or a downstream task to directly talk to slack • Attach a schedule to the flow to fit your desired run cadence https://docs.prefect.io/core/concepts/schedules.html#overview
    p

    Preston Marshall

    2 years ago
    awesome, thanks