I am new to prefect and investigating several use-cases to assess if Prefect would be a good fit in the toolbox to build a team of engineers dealing with data integration between our company's API and our customers systems.
For some use-cases we have to periodically pull files from a remote SFTP server, do some manipulations with the file and some APIs then finally upload a sort of "response" file over the SFTP server.
I couldn't find something related to SFTP in the built-in tasks list https://docs.prefect.io/api/latest/tasks/function.htmlWhat would be the best way to deal with SFTP file exchange with prefect ? Should I just use a custom function task and a python sftp library or is there a more suitable task ?
4 months ago
Hi @Valentin Baert, welcome to Prefect! If you're just getting started doing your PoC, I'd recommend trying out Prefect 2.0About SFTP integration - Prefect 2.0 is pretty much just Python - you can use any sftp-library in Python from your tasks and flows e.g.:
• Paramiko• Pysftp (even easier)