• a

    Alex Izydorczyk

    9 months ago
    Any chance the Airbyte/Prefect integration session today was recorded/video is available? I missed the event and seems like you can no longer register
    a
    Anna Geller
    4 replies
    Copy to Clipboard
  • Tom Klein

    Tom Klein

    9 months ago
    just wanna give props to the support here, it's on a league of its own 💯
    Tom Klein
    Kevin Kho
    4 replies
    Copy to Clipboard
  • y

    Yusuf Khan

    8 months ago
    Hadn't seen this gem before: Got it when signing out
  • Tony Liberato

    Tony Liberato

    8 months ago
    Good morning, All. This is a question about security. Imagine I have prefect cloud running and its sending tasks to a few agent I have set up on my local subnet. Is there any risks involved with the cloud server being hacked and sending malicious code to my agents? We have a lot of auditors check this kind of thing in our environment and I want to know how to address this with our security team.
    Tony Liberato
    Anna Geller
    +2
    6 replies
    Copy to Clipboard
  • o

    Oliver Verran

    8 months ago
    Hi all, we are interested in using Prefect for ETL jobs at my firm. These particular ETL jobs run daily, have strict deadlines, and are considered critical for our business. So with that, I have some questions around the reliability of prefect server. Essentially, how can we avoid Prefect server being a single point of failure for the orchestration of flows? I appreciate there's probably a lot of approaches here, but as a scenario, let's suppose our server running prefect server had a hardware failure during a critical period - what could we do to mitigate this?
    o
    Anna Geller
    3 replies
    Copy to Clipboard
  • Maximilian Rausch

    Maximilian Rausch

    8 months ago
    Is there any step by step tutorials or example on how to set up an agent to run a flow on a windows laptop with a local git repository? I've read through the docs but I can't find a full example and googling isn't helping much.
    Maximilian Rausch
    Kevin Kho
    12 replies
    Copy to Clipboard
  • Patrick Wyatt

    Patrick Wyatt

    8 months ago
    I'm aiming to copy database backups or snapshots from AWS RDS (MariaDb) into S3 so their data can be ingested with Prefect flows, and I'm kind of overwhelmed at the number of options: 1. Export RDS snapshots to S3 using AWS Lambda (scheduled with Cloudwatch Events) 2. AWS Glue to S3 3. AWS Data Migration Service to S3 4. AWS Backup to S3 5. AWS Data Pipeline to S3 My leading candidate is Lambda, but I thought I'd post this and see if anyone had suggestions. Thanks!
    Patrick Wyatt
    Anna Geller
    2 replies
    Copy to Clipboard
  • r

    Rehan Razzaque Rajput

    8 months ago
    Hi, We're thinking about using prefect for writing our "Flows". For very large workloads, we're looking for a distributed solution and we're so happy that prefect has such a smooth integration with dask. 😃 However, we also have workloads that are extremely light and require sub-second latency. We performed some benchmarks internally and realized that compared to bare-metal python, prefect has an overhead of about 50ms - which is okay for us. But, we were thinking that as prefect grows, is there a possibility that perhaps this overhead would grow too big (in seconds)? Ultimately, this is more of a question regarding the prefect philosophy. Is prefect going to concentrate only on large workloads that require throughput while sacrificing the latency for small workloads? Or, is it going to keep a balance between both worlds? Thanks!
    r
    Kevin Kho
    6 replies
    Copy to Clipboard
  • Kevin Kho

    Kevin Kho

    7 months ago
    Hi friends! I am a co-chair for the SciPy Conference Data Lifecycle track this year. Our goal is to introduce industry best practices to scientific computing. If you use Prefect (or any other data tools), please consider submitting an abstract to present. Always happy to chat if there are any questions 🙂
  • Olivér Atanaszov

    Olivér Atanaszov

    7 months ago
    Hi, I keep getting "The replica workerpool0-0 exited with a non-zero status of 127" in the console using the Vertex agent. Looking at the flow logs, I see the job submitted to Vertex, then after a few minutes I get "Rescheduled by a Lazarus process. This is attempt 1." and it just keeps retrying. Anyone had similar issues?
    Olivér Atanaszov
    Kevin Kho
    +1
    7 replies
    Copy to Clipboard