Is anyone going to KubeCon in LA next week? Prefect sponsored a booth and we’d love to meet you guys. Feel free to stop by and chat with us!

hey all! I recently wrote something with Ville Tuulos (metaflow/outerbounds) about the maturity of the MLOps stack(s). Would be very interested in any feedback! https://www.oreilly.com/radar/mlops-and-devops-why-data-makes-it-different/

Great blog from @Madison Schott. Great to have you as part of our community. https://towardsdatascience.com/is-dbt-the-future-of-analytics-d6ff93cbb20c

Whatever random generator package for upcoming flow runs is 🔥 Got me thinking about an arrogant 🐙 and a sassy 🦜

Hi all 🙂 Happy Friday! We posted a podcast with the amazing Prefect founder Jeremiah Lowin today talking about the early founding story & his learnings on this journey. Wanted to share in case folks were interested! https://anchor.fm/ossstartuppodcast/episodes/E10-Prefect---Open-Source-Data-Flow-Automation-e19ao4o

Just wanted to thank @Kevin Kho and @Anna Geller for always being helpful and providing quick and useful answers! 🙌 🦜

To our American community members: hope you have a great Thanksgiving holiday and if you’re traveling, please send us pictures of Marvin on the road! marvin duck If you’re not traveling, pictures of Marvin at home are welcome, too. marvin …in fact, we’ll accept any picture of Marvin. 🤷 Please excuse the Prefect team if we aren’t up to our usual lightning-fast response time over the next couple of days — we’re recharging and more than anything, grateful for the opportunity to work with you all!

(Photo credit @Sarah Moses)

One of the best reasons I have for working with Prefect is the fact that you discover a bunch of animals you did not know about. Today, the kakapo.

Been doing prefect stuff for a week, now "perfect" has become this other weird word

Any chance the Airbyte/Prefect integration session today was recorded/video is available? I missed the event and seems like you can no longer register

just wanna give props to the support here, it's on a league of its own 💯

Hadn't seen this gem before: Got it when signing out

Good morning, All. This is a question about security. Imagine I have prefect cloud running and its sending tasks to a few agent I have set up on my local subnet. Is there any risks involved with the cloud server being hacked and sending malicious code to my agents? We have a lot of auditors check this kind of thing in our environment and I want to know how to address this with our security team.

Hi all, we are interested in using Prefect for ETL jobs at my firm. These particular ETL jobs run daily, have strict deadlines, and are considered critical for our business. So with that, I have some questions around the reliability of prefect server. Essentially, how can we avoid Prefect server being a single point of failure for the orchestration of flows? I appreciate there's probably a lot of approaches here, but as a scenario, let's suppose our server running prefect server had a hardware failure during a critical period - what could we do to mitigate this?

Is there any step by step tutorials or example on how to set up an agent to run a flow on a windows laptop with a local git repository? I've read through the docs but I can't find a full example and googling isn't helping much.

I'm aiming to copy database backups or snapshots from AWS RDS (MariaDb) into S3 so their data can be ingested with Prefect flows, and I'm kind of overwhelmed at the number of options: 1. Export RDS snapshots to S3 using AWS Lambda (scheduled with Cloudwatch Events) 2. AWS Glue to S3 3. AWS Data Migration Service to S3 4. AWS Backup to S3 5. AWS Data Pipeline to S3 My leading candidate is Lambda, but I thought I'd post this and see if anyone had suggestions. Thanks!

Hi, We're thinking about using prefect for writing our "Flows". For very large workloads, we're looking for a distributed solution and we're so happy that prefect has such a smooth integration with dask. :) However, we also have workloads that are extremely light and require sub-second latency. We performed some benchmarks internally and realized that compared to bare-metal python, prefect has an overhead of about 50ms - which is okay for us. But, we were thinking that as prefect grows, is there a possibility that perhaps this overhead would grow too big (in seconds)? Ultimately, this is more of a question regarding the prefect philosophy. Is prefect going to concentrate only on large workloads that require throughput while sacrificing the latency for small workloads? Or, is it going to keep a balance between both worlds? Thanks!

Hi friends! I am a co-chair for the SciPy Conference Data Lifecycle track this year. Our goal is to introduce industry best practices to scientific computing. If you use Prefect (or any other data tools), please consider submitting an abstract to present. Always happy to chat if there are any questions 🙂

Hi, I keep getting "The replica workerpool0-0 exited with a non-zero status of 127" in the console using the Vertex agent. Looking at the flow logs, I see the job submitted to Vertex, then after a few minutes I get "Rescheduled by a Lazarus process. This is attempt 1." and it just keeps retrying. Anyone had similar issues?

This is a bit of a stretch I suppose, has anybody run MPI parallel jobs in k8s as a prefect task?

Heya is there a standard/guidance for passing in parameter structs between tasks? Ideally I’d define a struct via

@dataclasses.dataclass

or maybe

pydantic

and be able to pass around all attributes of the struct easily. Are people just using json or something?

Hi, I'm trying to use GCS storage with pickling my flow, but I get:

Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'utils\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')

In my flow definition, I'm trying to import from this package called

utils

but apparently cloudpickle did not take care of these imports. I've checked this guide but couldn't figure the canonical way of composing flows using imports from other packages/modules: https://docs.prefect.io/orchestration/flow_config/storage.html#pickle-vs-script-based-storage

Heya - google fu is failing me - has somebody already made an integration with prefect

Results

using mlflow as the storage layer?

It’s news to me numpy as an interactive demo environment in the docs

This is a really out there question, but have there been any thoughts about a mobile app to view the cloud ui? 🙂

What a date! 2022-02-22 anyone getting married or proposing today? Any crazy announcements? 😄

Hello, Prefect experts. We are bootstrapping our analytics stack. I was evaluating different modern solutions. We have production data in SQL database. We want to work on some analytics cleanup & transformation and ship to PowerBI. Based on the research I have done, it looks to me the viable solution for us will be dbt + warehouse(snowflake) + visualization(power bi). If that is the case, it sounds like we do not need prefect to orchestrate python based tasks. Do you have case study that is applying prefect to analytics pipeline achitecture? Hope someone can give me some direction.

Does anyone have any recommendations for Sublime Text package plugins relevant to Python. I only have python-black installed at the moment.

Hello, I am trying to run a prefect flow triggered by when there is a change in a github repository that is NOT the repo that the prefect flow lives in. TLDR: pull request on repository A triggers prefect flow living in Repository B. This is the closest I've come to figuring this out: https://docs.prefect.io/core/pins/pin-08-listener-flows.html Has anyone done this before or have instructions?