Hi, We are exploring alternative options to Airflo...
# best-practices
s
Hi, We are exploring alternative options to Airflow and were wondering if Prefect is compatible with build systems. We use bazel and some of our scripts are only visible when run through bazel. Can we configure flows to run through a build system?
a
1. Are you asking for Prefect 1.0 or 2.0? 2. What are you trying to do in your build pipeline? Can you explain your use case from business perspective? Forget about Prefect and bazel for a moment
s
I’m asking for Prefect 1.0. It’s mainly an ETL pipeline. We ingest data every few hours but we found that to be less ideal. Airflow doesn’t support the overhead with ingesting data at smaller intervals.
a
We ingest data every few hours but we found that to be less ideal.
that's totally understandable are you already using Prefect or are you just getting started?
Prefect 2.0 supports even running your ETL in a real-time service if you need that level of data freshness in your ETL workloads I'd definitely encourage you to start with 2.0 unless you've already built everything for 1.0 (this doesn't seem so based on your message though)
s
I’m just getting started by trying to run one of our flows locally
I see. I’ll check it out
a
great to hear - this thread can help figure out which Prefect version to use and will hopefully clarify any confusion
s
Irrespective, a core requirement for us is the compatibility with build systems. Do you know if it is possible to run tasks through build systems?
a
The short answer is yes, you can authenticate with Prefect Cloud API and trigger some action from a build system - it depends on what you're trying to do. Most people use CI/CD systems to package their code dependencies and create deployments (or in 1.0 register flows) from a CI, rather than running flows on merge to a given branch, but if you want to you could start a flow run upon merge
123 Views