Want to learn how to create multiple deployments from a single Prefect Project in just a few steps? Check out ! 🚀

Hey all! Wanted to share my org's story on migrating from Civis (an analyst-friendly but limited orchestration tool) to Prefect. Here's my blog post about it. I cover the limitations of Civis vis a vis data engineering best practices and how Prefect improves our workflows. I cover what it looks like to migrate a script from Civis to Prefect. I get into details about how we set up our execution layer in AWS ECS based on the dataflowops tutorial with a few modifications. I include a sample repository with working template scripts. I also demo a modification to Prefect's concurrency implementation to support rate limiting API calls. I always appreciate seeing demos of how folks go about their implementation and hope this is helpful if you're looking into how to set up Prefect in AWS ECS or move off of a limited legacy orchestration stack!

Hello fellow database nerds, and also Prefect Cloud users! Earlier this week we published a write up on a sequence of Prefect Cloud incidents that occurred due to database memory issues; if you are curious to learn more about those incidents, or if you want to learn some surprising facts about operating Postgres at scale, then this post is for you! Enjoy: https://medium.com/the-prefect-blog/more-memory-more-problems-b182d7807e3c

On a similar note, I just spent way too many hours digging around in SQLAlchemy / SQLite internals and there’s a much more poorly written writeup about handling concurrent SQLite queries safely at https://github.com/PrefectHQ/prefect/pull/9594

Just finished an introduction to data orchestrators (featuring prefect and others): Your data team wants to start using a data orchestrator. Great! Here’s a quick primer to get you started - to prevent you from spending dozens of hours wrestling up orchestrator infrastructure you don’t need. https://www.finishslime.com/c/data-orchestrator-101

SQL is no longer the only way to work with data in your DB. If you want to use pandas or numpy to work with the data inside your warehouse, feel free to check out the public beta of Ponder. Just launched today and runs out of the box with Prefect! https://ponder.io/ponder-in-public-beta-data-science-in-your-data-warehouse/

I have found myself repeating a similar setup for multiple projects at work for prefect pipelines. I templated the one we use at my company and it has been super useful spinning up new projects. So I thought I would take a stab at a more generic open source version of it. This is a template to help quickly setup a repo using the tools you want to use. It is opinionated to how I like my repos structured, but hopefully will provide a good starting point for anyone looking to use it. It provides the following: • A multistage build Dockerfile • Github actions for linting, testing, and deploying • Poetry setup • Prefect project setup • Just command file I currently am only supporting Docker and Process workers with Github storage. If there is more interest though I could expand it? Hopefully this is useful to someone! https://github.com/ngriffiths13/prefect-poetry-setup

Hi folks, I open sourced a small toolkit library supreme-task which extends the default and beloved prefect

task

decorator! It mainly adds the following two features • runtime-type checking: this helps avoid reducing boilerplate code checking task input types at runtime • persistence of task inputs on failure: this curries the task decorator with a callable that will store the task inputs in case of a failure, helping to better debug issues Hopefully it is useful to others. Feedback is more than welcome !

I have begun consolidating and standardizing Prefect code that I reuse across projects. I decided to package them into my first Python package, prefecto. Learned a bunch of new tools on the way and would love some feedback!

Wrote a little script that I run in CI/CD that finds the folder we toss all the Flow files into, then makes a new folder for each one, places the script into it, then `cd`s into it so it can deploy it without uploading our entire repo for every script every time. The main benefit is that adding/editing a new flow is just a one-step process ("put

.py

file into the folder").

import os
import subprocess
from pathlib import Path
from shutil import copy

starting_dir = Path(
    "" # Put the folder with the Flows here
).resolve()
all_scripts = list(starting_dir.glob("*.py"))

for script in all_scripts:
    without_extension = str(script).split(".")[0]
    folder = without_extension.split("/")[-1]
    just_script = f"{folder}.py"
    folder_path = Path(starting_dir, folder)
    Path(folder_path).mkdir(parents=True, exist_ok=True)
    new_script_path = Path(folder_path, just_script)
    copy(script, new_script_path)
    os.chdir(folder_path)
    subprocess.run(["python", new_script_path])

Hi everyone, I would like to share some work I have done integrating with the NASA Earthdata platform 🌍🛰️ this integration was initiated as a capstone project during the latest PACC in London, and I would like to express my gratitude to the instructors who provided valuable help and motivation to finalize it! Please check it out—I hope you find it useful 🙂 GitHub: https://github.com/giorgiobasile/prefect-earthdata Docs: https://giorgiobasile.github.io/prefect-earthdata/ LinkedIn: https://www.linkedin.com/feed/update/urn:li:share:7082369580899123200/

Used some of the newer Cloud features (event webhooks, deployment triggers) to kick off flow runs in response to kafka messages. https://medium.com/the-prefect-blog/beyond-scheduling-event-driven-flows-with-prefect-b072edd06833

Hello All We are using prefect cloud and getting this error... can someone pls help to resolve this issue... ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

Hi all 🙂 Wanted to show how I deployed a worker in a docker environment while using Prefect Cloud. I am somewhat new to docker, so maybe someone can improve it, but I want to describe the challenges I face. What I wanted: Start a docker container which, upon initialization, logs in to my cloud account and automatically starts a Prefect worker. The problem: Prefect already has a docker image, but simply writing a Dockerfile like

FROM prefecthq/prefect:2-python3.11 

ARG PREFECT_TK

RUN prefect cloud login --key "${PREFECT_TK}" --workspace "mortenginnerupandreasenzohocom/homelab"

CMD prefect worker start --pool my-process-pool

does not work. The

CMD

line overwrites all the lines in the original prefect image, rendering it all useless. Appending the

CMD

to the

RUN

line with and

&&

does not work, as

RUN

is executed when the image is build, not deployed. Solution: Use multi stage builds. My Dockerfile looks likes this

# Install he prefect image and login.
FROM prefecthq/prefect:2-python3.11 AS builder

ARG PREFECT_TK

RUN prefect cloud login --key "${PREFECT_TK}" --workspace "mortenginnerupandreasenzohocom/homelab"

# Cant start worker from first stage, as CMD will override all CMD's in the original prefect image.
# Therefore, a second stage is added, where a CMD can be freely added without consequences for previous
# CMD's.
FROM builder AS starter

CMD prefect worker start --pool my-process-pool

and my compose.yaml looks like this:

services:
  worker:
    build:
      context: .
      args:
        PREFECT_TK: ${PREFECT_TK}  
    env_file:
      - .env

The prefect token is saved in a .env file. The image can then be build with command:

docker compose build

(sudo might be needed depending on how you installed docker)

Hi everyone! My name is Thierry and I'm a developer advocate for the open-source project Hamilton. While Prefect is an orchestration tool that helps run code on infra (macro), Hamilton is a micro-framework that help you write data transformations in a principled manner (think dbt for Python). A core challenge of building a workflow is determining how to break it down into tasks. As your project grows, complex you have to find the right trade-off between the complexity of your workflow structure and the complexity of each task. At the orchestration level, adding too many tasks can add overhead and be operationally costly. In contrast, Hamilton encourages very granular data transforms and automatically creates the execution DAG for you! I just published a blog post on how Prefect and Hamilton complement each other! By adding Hamilton to your Prefect task, you get easier to test and reuse data transformations. Also, it can nicely integrate with Prefect's Block and Artifact to improve the expressivity of your workflows and your data lineage. Happy to hear your thoughts and chat about this problem space!

If anyone uses Azure AKS and wants to give more granular permissions to their deployments, Azure Workload Identity might help: https://discourse.prefect.io/t/use-aks-workload-identity/3354. And if anyone has experience with customizing work pools, there is room for a little improvement. Azure workload identity hooks into the kubernetes concept of service accounts, and is fairly elegant.

Hey everyone! 👋 Just some big news to share on Airbyte’s side: our official Airbyte API and *Terraform Provider are both now available on Airbyte Open Source!* Here’s a more detailed article about the release.

@George Coyne presented on "Orchestrating Pandas Code with Prefect" at July's Professional Pandas Meetup, and the recording is now live! You can check it out here:

Anyone else running prefect via singularity on academic HPC? I'm using the Dask-MPI backend. https://github.com/jonititan/FDMSingularity

Hi everyone, just letting you know I have released prefect-planetary-computer, a collection of Prefect integrations with the Microsoft Planetary Computer. It provides a Credentials Block :key: to store and retrieve a subscription key and a Hub token, with convenience methods for flows and tasks interaction with the PC Data Catalog :earth_africa: and Dask Gateway :rocket: server, and examples of how it nicely integrates with prefect-dask and prefect-azure. Hope it is useful, check it out 🙂 GitHub: https://github.com/giorgiobasile/prefect-planetary-computer Documentation: https://giorgiobasile.github.io/prefect-planetary-computer/ LinkedIn: https://www.linkedin.com/feed/update/urn:li:share:7095787809063469056/ Twitter: https://twitter.com/giob12/status/1690021882970419201

System that builds a dataset containing visually-confirmed equipment losses for Russia and Ukraine. Built on Prefect, AWS, and Terraform. It refreshes daily and publishes the latest data to Kaggle. Future plans for it include • Apply OCR algorithms to find dates in timestamped images • Expand the evidence extraction to handle X (Twitter) posts • Enhance loss information with specific data about equipment • Abstract schema documentation generator to own library and improve features https://github.com/dominictarro/Borderlands https://www.kaggle.com/datasets/dominictarro/borderlands

Hey folks, , dlt, "data loading tool", the OSS python library for data loading with d*eclarative* interface and schema inference and evolution now supports .. 🚀 Loading directly to Athena and registering tables in Glue data catalog! Here's the athena doc page - we plan to add more support in the future based on your interest and usage - so use it and give feedback! Here are other destination pages in case you are interested Here's what dlt pipelines look like - as simple as they can be.

Hi, everyone! I just published a multiprocess task runner for Prefect: https://github.com/rpeden/prefect-multiprocess I started this little side project nearly a year ago when I wanted to learn more about Prefect internals. It was primitive, but it mostly worked. I recently revisited it and found it was broken after the concurrency changes in Prefect 2.8.3. So, I decided to give it a rewrite since I had some workflows I wanted to run in parallel without needing to deal with Dask or Ray. They're great when you need them, but I wanted something a little simpler. This task runner uses a

ProcessPoolExecutor

from the standard library, and doesn't install extra dependencies beyond what Prefect already comes with. It works quite well for my tasks, so I figured I'd share it in case it's useful for anyone else. I've added some notes and examples in the README, so please feel free to give it a try!

introducing 3 new fun animated emojis for your use: prefect build prefect spin prefect rocket

Hello dears I got an error when trying to install perfect-docker https://prefecthq.github.io/prefect-docker/ at the register step (prefect-env) PS C:\Training\prefect> prefect block register -m prefect_docker Traceback (most recent call last): File "C:\Ericsson\Training\prefect\prefect-env\Lib\site-packages\prefect\cli\_utilities.py", line 41, in wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect\utilities\asyncutils.py", line 255, in coroutine_wrapper return call() ^^^^^^ File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 383, in call return self.result() ^^^^^^^^^^^^^ File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 283, in result return self.future.result(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 169, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\concurrent\futures\_base.py", line 401, in __get_result raise self._exception File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect\_internal\concurrency\calls.py", line 346, in _run_async result = await coro ^^^^^^^^^^ File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect\cli\block.py", line 135, in register imported_module = import_module(name=module_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1204, in _gcd_import File "<frozen importlib._bootstrap>", line 1176, in _find_and_load File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked File "<frozen importlib._bootstrap_external>", line 940, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "C:\Ericsson\Training\prefect\prefect-env\Lib\site-packages\prefect_docker\__init__.py", line 2, in <module> from .host import DockerHost # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Training\prefect\prefect-env\Lib\site-packages\prefect_docker\host.py", line 10, in <module> class _ContextManageableDockerClient(docker.DockerClient): ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'docker' has no attribute 'DockerClient' An exception occurred. (prefect-env) PS C:\Training\prefect> python Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 054537) [MSC v.1934 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

>>

Hi, I have been been working a bit on and off on getting prefect to work at home on a remote server with an otherwise minimal set up. In my reply I have posted a note on my experience. I dont have anywhere to post it properly, so its just pasted in here as text. Wall of text in the reply 🙂

Hi all! Sharing some work on using Pub/Sub messages to trigger Prefect flows! I chose to use Memphis as a message broker. My use case was to gather all our automations in one tool, we use Prefect for our data processing pipelines, GitLab CI/CD, and other internal tools for databases, front-end, ect... Now we integrate with Memphis to handle all kind of events in one place and automate all executions from new data processed to ML model training and frontend updates! Code : https://discourse.prefect.io/t/using-a-message-broker-to-trigger-flow-runs/3485

Are there any Julia users on here? I prefer it for analysis workflows, and made this to help me interoperate: https://mahiki.github.io/PrefectInterfaces.jl/stable/

hi community came across this case study about Jellyfish moving from celery to prefect https://assets.ctfassets.net/gm98wzqotmnx/7mUU9hzyzT6W86ITksz0Ev/af32716a6ebaa78fe35f2058999bc0bf/jellyfish-case-study.pdf looking for more detailed feedbacks on similar migration exercice 🙏

I'm not sure if it'll be useful to anyone other than me, but I just published a Prefect API client for .NET: https://github.com/rpeden/prefect-dotnet-client I use it in C# and F# and it works nicely there. But if you choose to (or have to) use VB.NET, it should work fine there too 🙂. It's all auto-generated using OpenAPI generator. It targets .NET Standard 2.0 so it should work in anything from a crufty LOB app that will be stuck on .NET Framework 4.6 until the end of time to a bleeding-edge AOT-compiled .NET 8 app. Right now, it only works with endpoints available in both Prefect OSS and Prefect Cloud - I did a lot of processing of Prefect's autogenerated OpenAPI spec to make it friendly to the OpenAPI generator and I just I haven't had time to do the cloud-only endpoints yet. Regardless, the client is pretty useful as-is. I mostly use it to work with flows, flow runs, and logs and haven't run into any issues. The repo includes a pretty good set of autogenerated docs but if you decide to give it a try and have any questions, please feel free to open a discussion on GitHub.