Valentin Baert

05/17/2022, 12:13 PM
Hello, I am new to prefect and investigating several use-cases to assess if Prefect would be a good fit in the toolbox to build a team of engineers dealing with data integration between our company's API and our customers systems. For some use-cases we have to periodically pull files from a remote SFTP server, do some manipulations with the file and some APIs then finally upload a sort of "response" file over the SFTP server. I couldn't find something related to SFTP in the built-in tasks list What would be the best way to deal with SFTP file exchange with prefect ? Should I just use a custom function task and a python sftp library or is there a more suitable task ?

Anna Geller

05/17/2022, 12:30 PM
Hi @Valentin Baert, welcome to Prefect! If you're just getting started doing your PoC, I'd recommend trying out Prefect 2.0 About SFTP integration - Prefect 2.0 is pretty much just Python - you can use any sftp-library in Python from your tasks and flows e.g.: • ParamikoPysftp (even easier)

Valentin Baert

05/17/2022, 12:32 PM
👍 1