Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hey, everyone! I've been working on a Data Engineering project using Prefect has the backbone of the project. This project focus on the Premier League where I am building a Streamlit dashboard packaged into a container that is hosted on Google Cloud's Cloud Run.

<https://storage.googleapis.com/premier-league/streamlit_demo_2.10.1.gif>

• Check the out the <https://streamlit.digitalghost.dev|dashboard>!
• <https://github.com/digitalghost-dev/premier-league|GitHub Repository>

gif demo is awesome! thanks for sharing :thank_you:

<@U05V1JY9PU1> Just curious why are you using a Cloud Scheduler for the Football API on the side?

Also, any reason why you have three separate Data pipelines instead of just 1?

<@U03F9TT4YG5> I am using Cloud Scheduler because it triggers the docker container I have hosted on Cloud Run as a job. Without it, I’d have to manually run it each time. I believe I could use Prefect to schedule the jobs but I had set this up way before I started using Prefect. I might have Prefect trigger it but as of right now, it’s fine.

As for why there are 3 data pipelines is because of organization. This being a personal project, I am experimenting with all sorts of technologies so these pipelines may not make efficient sense but it gives me the opportunity to learn.