https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
random
  • k

    Kevin Kho

    10/06/2021, 7:30 PM
    Is anyone going to KubeCon in LA next week? Prefect sponsored a booth and we’d love to meet you guys. Feel free to stop by and chat with us!
    :upvote: 4
    s
    1 reply · 2 participants
  • h

    hugo bowne-anderson

    10/19/2021, 10:53 PM
    hey all! I recently wrote something with Ville Tuulos (metaflow/outerbounds) about the maturity of the MLOps stack(s). Would be very interested in any feedback! https://www.oreilly.com/radar/mlops-and-devops-why-data-makes-it-different/
    :marvin: 6
    k
    1 reply · 2 participants
  • a

    Andrew Black

    10/22/2021, 3:38 PM
    Great blog from @Madison Schott. Great to have you as part of our community. https://towardsdatascience.com/is-dbt-the-future-of-analytics-d6ff93cbb20c
    🚀 4
    :marvin: 3
    🎉 2
    m
    k
    3 replies · 3 participants
  • d

    Donny Flynn

    11/01/2021, 11:50 PM
    Whatever random generator package for upcoming flow runs is 🔥 Got me thinking about an arrogant 🐙 and a sassy 🦜
    😂 4
    :toucan: 3
    :marvin-duck: 3
    🎯 3
    🐙 2
    :marvin: 6
  • a

    Amanda Robson

    11/05/2021, 11:33 PM
    Hi all 🙂 Happy Friday! We posted a podcast with the amazing Prefect founder Jeremiah Lowin today talking about the early founding story & his learnings on this journey. Wanted to share in case folks were interested! https://anchor.fm/ossstartuppodcast/episodes/E10-Prefect---Open-Source-Data-Flow-Automation-e19ao4o
    🚀 3
    :prefect: 4
    :marvin: 5
    j
    a
    2 replies · 3 participants
  • p

    Pierre Monico

    11/17/2021, 2:53 PM
    Just wanted to thank @Kevin Kho and @Anna Geller for always being helpful and providing quick and useful answers! 🙌 😛arty-parrot:
    :thank-you: 22
    💯 24
    :upvote: 22
    🙌 1
    k
    a
    +1
    3 replies · 4 participants
  • j

    Jeremiah

    11/24/2021, 4:24 PM
    To our American community members: hope you have a great Thanksgiving holiday and if you’re traveling, please send us pictures of Marvin on the road! :marvin-duck: If you’re not traveling, pictures of Marvin at home are welcome, too. :marvin: …in fact, we’ll accept any picture of Marvin. 🤷 Please excuse the Prefect team if we aren’t up to our usual lightning-fast response time over the next couple of days — we’re recharging and more than anything, grateful for the opportunity to work with you all!
    🧗‍♀️ 2
    :marvin: 5
    ⛰️ 3
    ❤️ 12
    j
    2 replies · 2 participants
  • j

    Jeremiah

    11/24/2021, 4:27 PM
    (Photo credit @Sarah Moses)
    😍 4
    :marvin-duck: 2
  • s

    Sylvain Hazard

    11/29/2021, 11:09 AM
    One of the best reasons I have for working with Prefect is the fact that you discover a bunch of animals you did not know about. Today, the kakapo.
    :party-parrot: 1
    🐦 13
    a
    b
    +2
    7 replies · 5 participants
  • t

    Tilak Maddy

    12/06/2021, 4:29 PM
    Been doing prefect stuff for a week, now "perfect" has become this other weird word
    😆 13
    🙌 1
    :marvin: 1
    💯 3
  • a

    Alex Izydorczyk

    12/15/2021, 12:28 AM
    Any chance the Airbyte/Prefect integration session today was recorded/video is available? I missed the event and seems like you can no longer register
    a
    a
    4 replies · 3 participants
  • t

    Tom Klein

    12/16/2021, 4:33 PM
    just wanna give props to the support here, it's on a league of its own 💯
    :thank-you: 6
    🙌 18
    k
    j
    4 replies · 3 participants
  • y

    Yusuf Khan

    01/04/2022, 9:29 PM
    Hadn't seen this gem before: Got it when signing out
    :marvin: 14
  • t

    Tony Liberato

    01/12/2022, 2:48 PM
    Good morning, All. This is a question about security. Imagine I have prefect cloud running and its sending tasks to a few agent I have set up on my local subnet. Is there any risks involved with the cloud server being hacked and sending malicious code to my agents? We have a lot of auditors check this kind of thing in our environment and I want to know how to address this with our security team.
    a
    g
    +1
    6 replies · 4 participants
  • o

    Oliver Verran

    01/18/2022, 12:07 PM
    Hi all, we are interested in using Prefect for ETL jobs at my firm. These particular ETL jobs run daily, have strict deadlines, and are considered critical for our business. So with that, I have some questions around the reliability of prefect server. Essentially, how can we avoid Prefect server being a single point of failure for the orchestration of flows? I appreciate there's probably a lot of approaches here, but as a scenario, let's suppose our server running prefect server had a hardware failure during a critical period - what could we do to mitigate this?
    a
    3 replies · 2 participants
  • m

    Maximilian Rausch

    01/19/2022, 3:33 PM
    Is there any step by step tutorials or example on how to set up an agent to run a flow on a windows laptop with a local git repository? I've read through the docs but I can't find a full example and googling isn't helping much.
    k
    12 replies · 2 participants
  • p

    Patrick Wyatt

    01/20/2022, 7:37 AM
    I'm aiming to copy database backups or snapshots from AWS RDS (MariaDb) into S3 so their data can be ingested with Prefect flows, and I'm kind of overwhelmed at the number of options: 1. Export RDS snapshots to S3 using AWS Lambda (scheduled with Cloudwatch Events) 2. AWS Glue to S3 3. AWS Data Migration Service to S3 4. AWS Backup to S3 5. AWS Data Pipeline to S3 My leading candidate is Lambda, but I thought I'd post this and see if anyone had suggestions. Thanks!
    a
    2 replies · 2 participants
  • r

    Rehan Razzaque Rajput

    01/21/2022, 5:01 PM
    Hi, We're thinking about using prefect for writing our "Flows". For very large workloads, we're looking for a distributed solution and we're so happy that prefect has such a smooth integration with dask. :) However, we also have workloads that are extremely light and require sub-second latency. We performed some benchmarks internally and realized that compared to bare-metal python, prefect has an overhead of about 50ms - which is okay for us. But, we were thinking that as prefect grows, is there a possibility that perhaps this overhead would grow too big (in seconds)? Ultimately, this is more of a question regarding the prefect philosophy. Is prefect going to concentrate only on large workloads that require throughput while sacrificing the latency for small workloads? Or, is it going to keep a balance between both worlds? Thanks!
    k
    6 replies · 2 participants
  • k

    Kevin Kho

    01/31/2022, 7:24 PM
    Hi friends! I am a co-chair for the SciPy Conference Data Lifecycle track this year. Our goal is to introduce industry best practices to scientific computing. If you use Prefect (or any other data tools), please consider submitting an abstract to present. Always happy to chat if there are any questions 🙂
    🎉 6
    🚀 6
    ❤️ 2
  • o

    Olivér Atanaszov

    02/04/2022, 2:37 PM
    Hi, I keep getting "The replica workerpool0-0 exited with a non-zero status of 127" in the console using the Vertex agent. Looking at the flow logs, I see the job submitted to Vertex, then after a few minutes I get "Rescheduled by a Lazarus process. This is attempt 1." and it just keeps retrying. Anyone had similar issues?
    k
    a
    7 replies · 3 participants
  • e

    Evan Curtin

    02/04/2022, 3:13 PM
    This is a bit of a stretch I suppose, has anybody run MPI parallel jobs in k8s as a prefect task?
    a
    k
    10 replies · 3 participants
  • e

    Evan Curtin

    02/08/2022, 5:09 PM
    Heya is there a standard/guidance for passing in parameter structs between tasks? Ideally I’d define a struct via
    @dataclasses.dataclass
    or maybe
    pydantic
    and be able to pass around all attributes of the struct easily. Are people just using json or something?
    a
    1 reply · 2 participants
  • o

    Olivér Atanaszov

    02/10/2022, 4:57 PM
    Hi, I'm trying to use GCS storage with pickling my flow, but I get:
    Failed to load and execute Flow's environment: FlowStorageError('An error occurred while unpickling the flow:\n ModuleNotFoundError("No module named \'utils\'")\nThis may be due to a missing Python module in your current environment. Please ensure you have all required flow dependencies installed.')
    In my flow definition, I'm trying to import from this package called
    utils
    but apparently cloudpickle did not take care of these imports. I've checked this guide but couldn't figure the canonical way of composing flows using imports from other packages/modules: https://docs.prefect.io/orchestration/flow_config/storage.html#pickle-vs-script-based-storage
    k
    4 replies · 2 participants
  • e

    Evan Curtin

    02/11/2022, 2:33 PM
    Heya - google fu is failing me - has somebody already made an integration with prefect
    Results
    using mlflow as the storage layer?
    k
    6 replies · 2 participants
  • k

    Kevin Kho

    02/11/2022, 2:47 PM
    It’s news to me numpy as an interactive demo environment in the docs
    👀 1
    e
    2 replies · 2 participants
  • l

    Leon Kozlowski

    02/21/2022, 4:20 PM
    This is a really out there question, but have there been any thoughts about a mobile app to view the cloud ui? 🙂
    k
    e
    +2
    6 replies · 5 participants
  • a

    Anna Geller

    02/22/2022, 12:28 PM
    What a date! 2022-02-22 anyone getting married or proposing today? Any crazy announcements? 😄
    😆 8
    j
    c
    +1
    3 replies · 4 participants
  • h

    Hui Huang

    02/25/2022, 7:26 PM
    Hello, Prefect experts. We are bootstrapping our analytics stack. I was evaluating different modern solutions. We have production data in SQL database. We want to work on some analytics cleanup & transformation and ship to PowerBI. Based on the research I have done, it looks to me the viable solution for us will be dbt + warehouse(snowflake) + visualization(power bi). If that is the case, it sounds like we do not need prefect to orchestrate python based tasks. Do you have case study that is applying prefect to analytics pipeline achitecture? Hope someone can give me some direction.
    👀 1
    c
    a
    4 replies · 3 participants
  • a

    Andrew Huang

    02/27/2022, 6:57 PM
    Does anyone have any recommendations for Sublime Text package plugins relevant to Python. I only have python-black installed at the moment.
    a
    k
    +2
    7 replies · 5 participants
  • a

    Apoorva Desai

    02/28/2022, 10:09 PM
    Hello, I am trying to run a prefect flow triggered by when there is a change in a github repository that is NOT the repo that the prefect flow lives in. TLDR: pull request on repository A triggers prefect flow living in Repository B. This is the closest I've come to figuring this out: https://docs.prefect.io/core/pins/pin-08-listener-flows.html Has anyone done this before or have instructions?
    k
    a
    4 replies · 3 participants
Powered by Linen
Title
a

Apoorva Desai

02/28/2022, 10:09 PM
Hello, I am trying to run a prefect flow triggered by when there is a change in a github repository that is NOT the repo that the prefect flow lives in. TLDR: pull request on repository A triggers prefect flow living in Repository B. This is the closest I've come to figuring this out: https://docs.prefect.io/core/pins/pin-08-listener-flows.html Has anyone done this before or have instructions?
k

Kevin Kho

02/28/2022, 10:11 PM
Hi @Apoorva Desai, our paradigm around this is triggering the Flow like this. You can hit the Prefect API with
Client.create_flow_run
or the
create_flow_run
task
Or you can even use a Python request
a

Anna Geller

02/28/2022, 10:25 PM
could you explain the problem that you try to solve a bit more? if you would like to just execute code from another repository, you could use pygit2:
@task(name="Clone DBT repo")
def pull_dbt_repo(repo_url: str, branch: str = None):
    pygit2.clone_repository(url=repo_url, path=DBT_PROJECT, checkout_branch=branch)
but if you are trying to build a CI/CD pipeline, you would usually register a flow on merge to e.g. master branch rather than triggering a flow
a

Apoorva Desai

03/01/2022, 8:56 AM
Thank you!
👍 1
View count: 6