Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello, I'm Patrick Wyatt, an online game developer in the Seattle area, and co-founder of One More Game. I work on data analytics in my "spare time" when I'm not doing all the myriad tasks related to developing &amp; publishing a game. We're pre-launch now so only have a few tens of users playing our game. Our data volume is quite low -- hundreds of records per day, not millions ... yet. I've really enjoyed building a pipeline to export JSON data from our game &amp; backend servers -&gt; AWS Kinesis Firehose -&gt; S3 data lake, and then using Prefect to manage flows that ingest and process the data in Firehose.

Hi <@U02SVAFHCRW>, that’s quite a stack! What is your current game?

Howdy Kevin -- we haven't announced yet, sorry -- it's still under wraps

Welcome to the community <@U02SVAFHCRW>, look forward to playing on announce. You've got a good set of PC gamers here (and one or two console <@U01SRTRJC0Y>)

Hey <@U02SVAFHCRW> :wave: I’m one of those PC gamers! Welcome to the community

Welcome <@U02SVAFHCRW>! Looking forward to the official announcement :wave:

Great <@U02SVAFHCRW>, we’d love to learn more about how you’re implementing that tech stack and improvements you’re seeing. Welcome!

<@U02A9Q0UMQ9> -- this is a green-field project for my new company, and the last time I built something like this was some years ago, so I guess the biggest improvement is that I was able to utilize solutions like Prefect, Firebolt, Kinesis and S3 instead of building bespoke solutions using on-prem servers!

The code I've written is a combination of C#/Kotlin/JavaScript for the game services that need to export data to Kinesis, and Python/SQL for Prefect ingestion and data-processing, of course.

The project is deployed/automated using Terraform to configure build pipelines on Buildkite for CI/CD; and deployed in Docker containers running in Nomad. So there's a fair bit of infra work to make this all function.

If you have some specific questions I'm glad to try to answer.

Cheers!

Thanks for sharing <@U02SVAFHCRW>, and welcome to the Prefect community!