Hi Everyone 🙂,
I recently built an End-to-End Data Engineering project for analyzing Chicago's Divvy Bikeshare dataset, I used the following techstack for building this project.
• Google Cloud Storage (GCS) - as Data Lake
• Google BigQuery - for Data Warehouse
• Terraform - as Infrastructure-as-Code (IaC) tool
• Prefect - for orchestration
• dbt - for transformation and data modeling
• Google Looker studio - for visualizations
please do check it out and don't forget to star the GitHub repo, and let me know what you think.
link to repository : https://github.com/Fozan-Talat/divvy-bikeshare-de-project
05/02/2023, 2:58 PM
I like a lot of what I see in that repo, but I am curious. You commented out the dbt portion of your Prefect flow due to errors you were getting. What errors?
05/02/2023, 3:12 PM
This is really cool! Thanks for sharing
05/02/2023, 3:29 PM
@Kevin Ruprecht Finished in state Failed('Flow run encountered an exception. RuntimeError: PID 17072 failed with return code 1.\n'),
I also posted in the channel prefect-dbt but got no response, so I completed it without triggering dbt from prefect.
05/02/2023, 3:36 PM
Got it. thanks. I ask because I've been having problems trying to use Prefect blocks for the dbt project / profile paths. That error is not super-specific, but I think I got it at one point or another.