https://prefect.io logo
Title
s

Shriram Holla

05/30/2022, 2:11 PM
Hi, We are exploring alternative options to Airflow and were wondering if Prefect is compatible with build systems. We use bazel and some of our scripts are only visible when run through bazel. Can we configure flows to run through a build system?
a

Anna Geller

05/30/2022, 2:16 PM
1. Are you asking for Prefect 1.0 or 2.0? 2. What are you trying to do in your build pipeline? Can you explain your use case from business perspective? Forget about Prefect and bazel for a moment
s

Shriram Holla

05/30/2022, 2:22 PM
I’m asking for Prefect 1.0. It’s mainly an ETL pipeline. We ingest data every few hours but we found that to be less ideal. Airflow doesn’t support the overhead with ingesting data at smaller intervals.
a

Anna Geller

05/30/2022, 2:47 PM
We ingest data every few hours but we found that to be less ideal.
that's totally understandable are you already using Prefect or are you just getting started?
Prefect 2.0 supports even running your ETL in a real-time service if you need that level of data freshness in your ETL workloads I'd definitely encourage you to start with 2.0 unless you've already built everything for 1.0 (this doesn't seem so based on your message though)
s

Shriram Holla

05/30/2022, 2:51 PM
I’m just getting started by trying to run one of our flows locally
I see. I’ll check it out
a

Anna Geller

05/30/2022, 2:58 PM
great to hear - this thread can help figure out which Prefect version to use and will hopefully clarify any confusion
s

Shriram Holla

05/30/2022, 3:15 PM
Irrespective, a core requirement for us is the compatibility with build systems. Do you know if it is possible to run tasks through build systems?
a

Anna Geller

05/30/2022, 3:57 PM
The short answer is yes, you can authenticate with Prefect Cloud API and trigger some action from a build system - it depends on what you're trying to do. Most people use CI/CD systems to package their code dependencies and create deployments (or in 1.0 register flows) from a CI, rather than running flows on merge to a given branch, but if you want to you could start a flow run upon merge