Hello, I'm new to Prefect and I'm wondering if I can use Prefect for my use case. I have a python script that extracts data from Google sheets and loads them into SnowFlake. A cron is used to schedule it running daily. Similarly, there are singer taps and targets orchestrated by the pipelinewise to load data from databases to SnowFlake. They are also scheduled by crons. The loaded data are then transformed using DBT and are scheduled much later than the above crons. Most of the examples I see in the Internet are workflows running python functions with task decorator.

Hi @Sumit Kumar Rai! Yes this sounds like a use case that Prefect can perfectly handle. We have a lot a task library that a lot of community members have contributed to. The Snowflake task and dbt task are commonly used. there is also a Google Sheets task. This can all be tied together with Prefect. Yes, you will have to write Python code but there is definitely a lot to start with.

@Kevin Kho Does that mean I have to ditch my python script that loads the Google Sheet and rewrite it as prefect task?

If you already have a Python script, all you have to do is wrap it in a function and apply the

@task

decorator to use it in Prefect. Prefect by design is non-invasive and cleanly wraps around your existing code.

What about the singer taps and targets that I'm using in pipelinewise? How can I use them in Prefect?

I am not familiar with this. Let me look at bit

This is helpful. But how can I make the pipelinewise cli available?

How are you running them at the moment? Through a UI?

What I meant was, in the case of dbt, I can pass dbt command and the

ShellTask

runs the command. But the dbt cli has to be somewhere, maybe it is already installed in the Prefect agent (I don't know).

I’m seeing this in the pipelinewise docs

pipelinewise run_tap --tap mysql_sample --target snowflake

If you use it like this, Prefect can just call it with the shell task.

Currently we are ok with the batch jobs for our use cases.

Yes you can make your own image and pass it to an agent. Only requirement is that prefect can be installed on it.

I noticed that GitHub tasks doesn't have clone tasks. How can I retrieve and use my dbt project that exists in the repo?

You can probably do it through the Docker container that you pass to run things on. Would that work for you?

If I pass the docker container, will the Prefect's DbtTask be in use?

It does but you would need to install

dbt

itself in the container.