I have been working a lot with setting up a developer workflow and deployment options lately, and created a demo project with one possible deployment pattern using Github Actions. Very much inspired by the github issue on the same topic. In short, this is one repo that contains flows that are deployed automatically and in parallel via the Github Action. Everything is dockerized so that the flows can be completely independent. So far this is in the trial-stage. I'm looking forward to exploring how to adapt this to Orion. https://github.com/radbrt/prefect_flows

http://www.tharwan.de/mlops-gridsearch.html we used prefect, sklearn and dask for easy grid searches in the hyper parameter space of Ml models. Overall a very enjoyable toolchain.

hey guys, am trying to run dbt job with prefect, but am always getting failed exit code 127. Any suggestion?

Alright @Marvin, do you like Christmas? 🎄 I set out to make a star with the radar plot. I figured it should be doable and would be a pretty cool visualisation. However it was difficult due to the unpredictability of which task lands where on the same level in the radar circle. Quick turnaround to new idea. I could use the same increasing amount of dependent task idea to make something that was a bit more consistent in form. Enter the Christmas tree, to kick off the very first challenge! By increasing the amount of dependent tasks at each level in the tree to create more branches I managed to get the tree structure plotted in Orion. I found that the plot extends to “3 o’clock” at first to then go down, in clockwise manner. This made my idea of creating a stem at the bottom hard, as it would be located at the upper part of the tree and not in the middle. I then realised that it would be possible to get a star or ball on the top of the tree which really topped it off I’d say! Hoping to see some cooler submissions in the week to come. Good luck and enjoy to anyone trying out the challenge 🙌

I also took a go at @Marvin's challenge :marvin-duck:. Although my flow's radar plot doesn't really look impressive, the experience leading up to it has been! I hacked together a simple flow that checks for new job postings on https://boards.greenhouse.io/prefect/, keeps track and when new ones come up, send their links to me via Telegram. I'll post the code in a reply in the thread (again, it was a hacked together quickly 😅 ) Note for the radar diagrams: left was my first flow run, and right the second run. The flow only notifies me when it found "new to me"-job posts. The second run skipped the sending step, because it already notified me of the current jobs. _What was

really

nice_: flows are so easy to define in Prefect 2.0! The conditional logic was done using a simple

if

statement. Not that 1.0's

case

is difficult, but 2.0's usage is super straight forward, making it even easier and more fun to use!

A new great blog post by @Jamie Zieziula 🔥 It will walk you through how to deploy the Docker agent on AWS EC2 using Terraform https://www.prefect.io/blog/announcing-the-terraform-module-to-deploy-the-prefect-docker-agent-on-aws/

message has been deleted

Some great new content by @Jean Luciano. https://www.prefect.io/blog/from-pdf-to-snowflake/

We’ve got both a blog and a companion video today on the Prefect and Great Expectations integration, both from our own @alex! Blog: https://www.prefect.io/blog/continuous-data-validation-with-great-expectations-and-prefect Video:

Dear all, I've written a post about how to deploy a Prefect Flow as Kubernetes Job at an Azure Kubernetes Service. A huge thanks to @Anna Geller: you helped a lot for making this post possible! https://www.patrickkoch.dev/posts/post_15/

Hi all - thanks again for @Henning Holgersen coming on PrefectLive. You can find PrefectLive recordings on our Youtube channel - check out !

New content on the Prefect blog this week! Get Integrated to Google Cloud Platform With prefect-gcp

What is the Modern Data Stack?

Hi there! Check out this cool guide on automating dbt exposures published today by @James Sopkin. This demonstrates how to automate the generation of your exposures using an example with dbt :dbt:, Tableau, and Github Actions. Thanks James! If you're interested in writing anything about Prefect, we'd love to feature you in our blog! You can reach out directly to me here on Slack or at chrisr@prefect.io.

Hi Prefectionists! Today seems to be a 😛refect: + :dbt: day! If you are using Prefect with dbt, we have another fantastic recipe you should try out. @alex has just released a full tutorial allowing you to easily rerun failed dbt models from a failed dbt DAG node. You only need to add one more

DbtShellTask

to your flow to take advantage of this functionality, making dbt DAG orchestration and recovery from failure that much easier! For more details on that, check this Discourse announcement. And to get directly to code, visit the Customer Success Recipes repository. Happy Engineering!

Fully dockerized prefect with preseeded deployments, queues, and an entrypoint that spins up agents and server. Auto deployed in a GitOps fashion to GCR (GAR). Cool!

Ever wonder how we got to the now of modern data tools? Where did dataflow automation come from? Do you want to reminisce about Informatica and Oracle (oh wait, maybe you're still using those...)? Do you yearn to understand why cron tortures you to this day? @justabill offers his creative writing style to A Brief History of Dataflow Automation - an entertaining read to start your morning.

Hello, here’s a quick blog post I wrote about integrating Sentry in Prefect Flows. Enjoy. https://source.coveo.com/2022/04/20/integrating-sentry-with-prefect-flows/

Hi all! I want to highlight our repo that is associated with an O'Reilly course that we helped create called 'Getting Started with Workflow Orchestration'. If you're just looking to schedule a recurring task, or you want to orchestrate your entire stack, this free repo is a good place to start.

New blog post by @Andrew Black about our partnership with Snowflake and the roadmap for 😛refect: + ❄️ integrations in Prefect 2.0 😛refect2: :cloud2:

There's been a great mix of content lately that this community may find helpful: • A guide from @Jean Luciano on how to deploy flows in a CI/CD fashion using Github Action • @Anna Geller’s workflow orchestration vs data orchestration writeup that can help you explain Prefect's purpose, to people who aren't familiar with Prefect! Happy reading!

In part 2 of @James Sopkin's social media series, he covers extracting data from Youtube videos using Prefect & the Youtube API. Read it here!

Was having this discussion on #prefect-community with @Anna Geller and other folks . Workflow orchestration makes me think about micro services , BPMN, Temporal, etc

Happy weekend all - wanted to highlight this recent community blog post covering running Prefect flows as Kubernetes jobs on Azure's Managed Kubernetes Service. Thanks for your contribution to the community @Patrick Koch! https://www.patrickkoch.dev/posts/post_15/

Hey guys! Recently, I was struggling with cloud infrastructure deployment and had the idea to create a tool. If you know how to code but are not familiar with clouds then this is a great way for you-just use Cloudy templates which are actually based on Pulumi. Let’s push “infrastructure as code” so it can be used by everyone! https://news.ycombinator.com/item?id=31313931 https://github.com/cloudytool/cloudy

message has been deleted

At Pocket (part of Mozilla) we recently open-sourced our Prefect repo: https://github.com/Pocket/data-flows One cool thing we do is to set the Flow Readme's during continuous deployment: https://github.com/Pocket/data-flows/blob/main/deploy/register_flows.py#L71-L94 (Our continuous deployment strategy is based on an approach from mousetree.)

Hello! I have being exploring Orion this week and I have put together a docker-compose to run Prefect 2.0 locally. I have tested it on mac and linux. The script runs prefect server, postgres, a docker agent, and minio to store flows. The start script configures a work queue and an example flow is included. Sharing in case someone finds it useful: https://github.com/fraibacas/prefect-orion

Hi all - today we're releasing the 2022 State of Workflow Orchestration report in conjunction with Gradient Flow. This was a survey of ~600 data engineers, data scientists and analysts focusing on the orchestration market. It measures use case, feature importance, job title of users, company size and industry, and more. You can read our blog summary here, and for the community we're posting the report directly in Slack hoping that you can make use of it. This may help you to explain what orchestration is, make a business case for dataflow automation tooling internally at your own organizations, or just be some pleasure reading 😉 Please feel free to share with your friends - they can download from this landing page.