Hi everyone and @Marvin , I have a use case to move/sync data between 2 databases short term, SQL server and Snowflake. I'm thinking to use Airbyte for this. Anyone know if Prefect integrates with PyAirbyte and could be deployed on say Prefect ECS workpools?

It seems there's no direct mention of a Prefect integration with PyAirbyte, but Prefect does support integrations with various third-party services, which you can explore here. You might need to create a custom task to interact with PyAirbyte using Prefect. As for deploying Prefect on ECS workpools, Prefect does support deployment on AWS ECS. You can set up a work pool of type ECS, and use either EC2 or Fargate as the capacity provider. The detailed guide for setting up ECS work pools can be found here. If you decide to proceed with using Prefect for your use case, you might need to create a custom integration with PyAirbyte, or check if the community has developed any unofficial integrations.

Hey Moe! Sounds like PyAirbyte is just a python package, so you should be able to use it within your Prefect workflows like any other package! Using it with ECS should be fine too. 👍

Thanks Bianca, I'm going to try it out