Hi Everyone š,
I recently built an End-to-End Data Engineering project for analyzing Chicago's Divvy Bikeshare dataset, I used the following techstack for building this project.
⢠Google Cloud Storage (GCS) - as Data Lake
⢠Google BigQuery - for Data Warehouse
⢠Terraform - as Infrastructure-as-Code (IaC) tool
⢠Prefect - for orchestration
⢠dbt - for transformation and data modeling
⢠Google Looker studio - for visualizations
please do check it out and don't forget to star the GitHub repo, and let me know what you think.
link to repository : https://github.com/Fozan-Talat/divvy-bikeshare-de-project
šŖ 11
ā¤ļø 10
:upvote: 6
š„ 5
š² 2
k
Kevin Ruprecht
05/02/2023, 2:58 PM
I like a lot of what I see in that repo, but I am curious. You commented out the dbt portion of your Prefect flow due to errors you were getting. What errors?
š 1
j
jawnsy
05/02/2023, 3:12 PM
This is really cool! Thanks for sharing
š 1
f
fozan talat
05/02/2023, 3:29 PM
@Kevin Ruprecht Finished in state Failed('Flow run encountered an exception. RuntimeError: PID 17072 failed with return code 1.\n'),
I also posted in the channel prefect-dbt but got no response, so I completed it without triggering dbt from prefect.
k
Kevin Ruprecht
05/02/2023, 3:36 PM
Got it. thanks. I ask because I've been having problems trying to use Prefect blocks for the dbt project / profile paths. That error is not super-specific, but I think I got it at one point or another.