Hi all, I'm not a data engineer 😄 .. rather on the data analysis and management side. Always been interested in related domain such as DE and figured to learn and get hands-on experience with personal projects. I stumbled upon Prefect while researching what I think I need to learn the basics, from data source to destination, with transformations in between, and automation/scheduling. First rookie question: can I use Prefect for transformations too? Or will I need dbt for that? Starting with babysteps, what I'm trying to accomplish is this (see pic). Nice meeting you all!!

Can you use Prefect for transformations? Absolutely. Is it the right tool for the job? As always, that depends on your circumstance. If you're familiar with dbt (since you mentioned it), I'd personally connect Prefect with dbt and let each tool do what it does best (orchestration and transform, respectively). But, since you can pretty much write any python code you want and run it in Prefect, the sky's the limit. If the data is just some simple work to "tinker" and learn, I've done work with pandas and polars with no issues. We're ramping up Prefect in our organization to manage all orchestration between our various tools (Fivetran, Airbyte, S3 loads, Snowflake tasks, eventually dbt, internal audits, batch jobs, etc.) so we have a single place to manage data workflows.

Yeah I'm familiar with dbt and its almost god-like status, but have zero experience with it. Since I want to learn both transformations in a pipeline fashion and some orchestration/scheduling, I figured, it would be great if I can do both in a single tool.