Valentin Baert

    Valentin Baert

    4 months ago
    Hello, I am new to prefect and investigating several use-cases to assess if Prefect would be a good fit in the toolbox to build a team of engineers dealing with data integration between our company's API and our customers systems. For some use-cases we have to periodically pull files from a remote SFTP server, do some manipulations with the file and some APIs then finally upload a sort of "response" file over the SFTP server. I couldn't find something related to SFTP in the built-in tasks list https://docs.prefect.io/api/latest/tasks/function.html What would be the best way to deal with SFTP file exchange with prefect ? Should I just use a custom function task and a python sftp library or is there a more suitable task ?
    Anna Geller

    Anna Geller

    4 months ago
    Hi @Valentin Baert, welcome to Prefect! If you're just getting started doing your PoC, I'd recommend trying out Prefect 2.0 About SFTP integration - Prefect 2.0 is pretty much just Python - you can use any sftp-library in Python from your tasks and flows e.g.: • Paramiko Pysftp (even easier)
    Valentin Baert

    Valentin Baert

    4 months ago
    Thanks