Have you considered taking the AWS Certified Machine Learning - Specialty exam? I took it a recently and couldn’t find many updated resources for it, so I wrote this post to help folks decide whether to take it and how to study for it.

cross-posting a new community post on getting started - thanks for sharing @anurag ambuja 🙌

Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This series of video will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection. • • View the full Prefect playlist.

How to get started with Prefect & GCP - agent deployed to a Compute Engine VM and flow runs to serverless Cloud Run containers - repo template with GitHub Actions automations included ✨

Hey everyone! If you’re looking to get into analytics engineering or sharpen your skills/best practices in 2023, I just launched my first ebook- The ABCs of Analytics Engineering. This is a guide to all of the major concepts and tools you need to know to be successful. There’s a chapter on data quality, orchestration, data pipelines, documentation, modularity, freshness, testing, and more! https://madisonmae.gumroad.com/l/learnanalyticsengineering

Happy new year, all! 🎉 If you’ve been around for a minute you know that at Prefect we add new features fast. Blink and you miss one. I made a recap of my ten favorite new enhancements since v2 GA five months ago. Check it out to see the highlights and let us know if you have other favorites. 🙂

This article and will teach you how to build a full-stack ML application with Pydantic and Prefect.

cross-posting: new Sentry block from the community ✨

Kudos to @Danilo Drobac for contributing this very detailed Prefect tutorial :gratitude-thank-you:

Thanks @Anna Geller, and @Jeff Hale for the proof reading and support on finishing it! I enjoyed writing this one, hopefully it helps somebody at least 🙂 The following part about deployment will be up in the next 10 days (planned).

I love the logger. How do I print the task’s inputs into the logger? Like: “Submitted task run ‘function(x=“Hello world”)’. I want to easily know which input it failed on

Giving writing a go.. here's one I made earlier regarding migrating flows from v1 to v2

New post on how to use Prefect with Fivetran 😛refect:

I had some fun with Prefect 2, Azure Container Instances and ARM templates this week. If anyone wants to try it out, I have a small writeup and a quickstart to create an agent on ACI, a key vault (complete with secrets), access to it from ACI, the necessary prefect blocks, and a flow to demonstrate. https://dev.to/radbrt/prefect-in-azure-container-instances-2jph https://github.com/radbrt/prefect_aci

Continuing the Azure theme, here's a quick guide on how to send Prefect logs to Azure Application Insights: https://discourse.prefect.io/t/how-to-send-prefect-logs-to-azure-application-insights/2202

Do you want to use Prefect in your data science project, but don’t know how to get started? My article How to Structure an ML Project for Reproducibility and Maintainability will show you to use a data science template to create an ideal project structure.

Hey folks - New Article on a Full Deployment of Prefect 2.0 with Prefect Cloud & GCP!

Reposting for visibility Anyone else on Cloud 2 get a Prefect Wrapped?

One year ago I opened the issue for arm64 support. Many thanks to ddelange and @Michael Adkins who made this possible with the latest prefect 2.7.9 release! I got the arm64 build running on my Raspberry Pi 4 (4GB RAM, 64Bit OS) using Prefect Cloud and running the prefect-agent in Portainer. @Anna Geller provided a great Dataflow Repository and I used the health check flow as the first flow on my pi. I am also looking forward to switch to AWS Graviton instances. Thanks to the whole prefect team for the amazing work! It is incredible to see how prefect evolved from 0.X ➡️ 1.X ➡️ 2.X. The Slack support as well as the stability and features of prefect are really awesome!

Hello Prefect community! Over the weekend, I dove into the Prefect-GCP library and decided to build a data pipeline with Prefect, GCP, and dbt. This was a great learning experience and I wanted to share my learnings, check out a quick walkthrough here - https://jackskylord.medium.com/sky-pipe-prefect-gcp-dbt-data-pipeline-80561107c38f

Many of you requested a diagram of Prefect concepts - we now have it! Big shoutout to @Taylor Curran for creating it 💙

My PyData NYC talk was just posted. Start at 9:38 if you want to hear how Prefect relates to Charles Babbage. 🙂

New post by @Andrew Huang on how to leverage the new prefect-openai collection to explain exception messages in plain English. Very cool use case, I highly recommend checking it out!

New post by @Nate about ELT using Airbyte, dbt, Snowflake, and of course... 😛refect:

Hi all ! I've been playing around with the idea using prefect in conjunction with JuiceFS for a distributed runtime environment. In essence the idea is to use JuiceFS to mount a distributed files system to K8s pods to act as a shared virtual filesystem. To simplify local development one simply has to manage a the runtime environment variables appropriately. In this example repo I attempt to illustrate a basic local configuration. Any ideas and comments would be much appreciated! https://github.com/CrispyCrafter/JuicyPrefect

Hello Prefect Community, I started working on a distributed task queue library a few months back. The library is available as a python package to install a start using : daskqueue - pypi package For all its greatness, Dask implements a central scheduler (basically a simple tornado event loop) involved in every decision, which can sometimes create a central bottleneck. This is a pretty serious limitation when trying to use Dask in high-throughput situations. Daskqueue is a small python library built on top of Dask and Dask Distributed that implements a very lightweight Distributed Task Queue. Daskqueue also implements persistent queues for holding tasks on disk and surviving Dask cluster restart. This could be used as a special Executor to enqueue work. I have experiment the limits of the

.map()

method where mapping on a large Iterable will bring the Dask cluster to a halt. I also wrote an article about implementation details: https://medium.com/@aminedirhoussi1/daskqueue-dask-based-distributed-task-queue-6fb95517dfea Hope you enjoy it, can't wait to hear about your feedback 😉 !

set the channel description: A place for sharing knowledge about Prefect, tips and tricks about working with data, and best practices about workflow orchestration and working with Python Prefect Recipes: https://github.com/PrefectHQ/prefect-recipes

Looking forward to the newest Prefect Certified grads showing us what they got! Thank you all for a great session. :catjam: