Hey, everyone! I've been working on a Data Engineering project using Prefect has the backbone of the project. This project focus on the Premier League where I am building a Streamlit dashboard packaged into a container that is hosted on Google Cloud's Cloud Run.
@Christian Sanchez Just curious why are you using a Cloud Scheduler for the Football API on the side?
John Kang
11/22/2023, 4:24 PM
Also, any reason why you have three separate Data pipelines instead of just 1?
c
Christian Sanchez
11/22/2023, 5:19 PM
@John Kang I am using Cloud Scheduler because it triggers the docker container I have hosted on Cloud Run as a job. Without it, I’d have to manually run it each time. I believe I could use Prefect to schedule the jobs but I had set this up way before I started using Prefect. I might have Prefect trigger it but as of right now, it’s fine.
As for why there are 3 data pipelines is because of organization. This being a personal project, I am experimenting with all sorts of technologies so these pipelines may not make efficient sense but it gives me the opportunity to learn.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.