https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-integrations
  • o

    Oscar Björhn

    11/01/2022, 3:04 PM
    @Oscar Björhn has joined the channel
  • r

    Rob Freedy

    11/01/2022, 3:10 PM
    @Rob Freedy has joined the channel
  • c

    Charleigh Liang

    11/01/2022, 3:10 PM
    @Charleigh Liang has joined the channel
  • s

    Stephen Herron

    11/01/2022, 3:11 PM
    @Stephen Herron has joined the channel
  • r

    redsquare

    11/01/2022, 3:11 PM
    @redsquare has joined the channel
  • c

    Claire Herdeman

    11/01/2022, 3:28 PM
    @Claire Herdeman has joined the channel
  • n

    Nate

    11/01/2022, 3:30 PM
    @Nate has joined the channel
  • j

    John Mizerany

    11/01/2022, 3:33 PM
    @John Mizerany has joined the channel
  • c

    Charleigh Liang

    11/01/2022, 5:41 PM
    Hi all! 👋 Friendly neighborhood PMM here; thought I'd kick this channel off with a quick poll/survey!
    :prefect: 4
    🚀 4
  • u

    user

    11/01/2022, 5:42 PM
    Which of the following tools do you use most often with your current data stack?
    r
    c
    • 2
    • 2
  • k

    Kalise Richmond

    11/03/2022, 9:22 PM
    🎉 prefect-gitlab integration just released! :marvin: This includes 2 blocks for GitLab credentials and GitLab Repository. https://github.com/PrefectHQ/prefect-gitlab
    🦊 6
    😀 5
    👍 6
    :blob-attention-gif: 4
  • b

    Bianca Hoch

    11/04/2022, 9:14 PM
    📣 Say hello to *prefect-discord*~! Brought to you by our very own @Ryan Peden 😛refect: ! "A collection containing a notification block making it easy to send messages and notifications from Prefect to a Discord channel." 👾 https://github.com/rpeden/prefect-discord
    :party-parrot: 4
    :gratitude-thank-you: 2
    🔥 7
    🎉 3
    🚀 3
    🙌 3
  • r

    Ricardo Gaspar

    11/08/2022, 4:35 PM
    hey there. Is there any plan for a
    openlineage
    integration? For ref: • https://openlineage.io/ • https://github.com/OpenLineage/OpenLineage/issues/81 • there was one for Prefect v1 https://github.com/OpenLineage/OpenLineage/pull/293 CC @limx0 @Anna Geller
    a
    b
    • 3
    • 7
  • t

    Tibs

    11/16/2022, 10:43 AM
    Hi everyone, we are migrating from prefect 1 to prefect 2, in our previous flows we are using sqlalchemy ORM to run queries on our database, would you recommend to switch to the prefect-sqlalchemy library and use plain SQL? (edite
    ✅ 1
    r
    • 2
    • 1
  • t

    Tibs

    11/17/2022, 1:31 PM
    Hi everyone, is there a way to specify secrets for an ECSTask block? Something similar to what can be done in the task definition? (see attached image from AWS docs)
    ✅ 1
    a
    • 2
    • 7
  • t

    Tibs

    11/22/2022, 4:50 PM
    Hi, I am looking for some help with prefect_sqlalchemy for how I could customize the sqlalchemy_execute task? Like giving a custom name, number of retries if it is possible
    n
    • 2
    • 2
  • j

    James Tisato

    11/27/2022, 11:28 PM
    Hi all. We use conda to create virtual environments for our Python code, and would like to do the same for running our flows. I noticed that the base
    prefect
    package is available on
    conda-forge
    but the integration packages are not. I raised https://github.com/conda-forge/prefect-feedstock/issues/179 about this but I wasn't sure if that was the right place for it so it would get noticed. Is there any guidance on whether you can make the Prefect-maintained integration packages available via conda as well as pip?
    r
    • 2
    • 1
  • t

    Tibs

    11/28/2022, 4:19 PM
    Hi everyone, we are having some issues with prefect on AWS. We use an S3 block to store (so using the s3fs), the we use boto3 in some of our tasks. The problem is, s3fs depends on an older botocore then our boto3 version, and we don't want to downgrade it. Does anyone have any recommendation on how to solve this? Will it be a problem in prefect_aws module?
    ✅ 1
    r
    • 2
    • 2
  • k

    Khuyen Tran

    11/29/2022, 4:19 PM
    Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This article will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection.
    🙌 3
    :party-parrot: 1
    :gratitude-thank-you: 1
  • c

    Chris Reuter

    11/29/2022, 11:21 PM
    x-posting here with some information from Fivetran on our new Collection!
    :gratitude-thank-you: 3
    🤠 3
    🎉 4
    🙌 3
    :fivetran: 1
  • m

    Michael Kahan

    11/30/2022, 10:04 PM
    Reposting a question here for this audience: Has anybody used
    prefect-airbyte
    to trigger syncs on Airbyte Cloud? Or is it only for open source hosted instances?
    c
    n
    • 3
    • 4
  • j

    James Tisato

    12/03/2022, 2:02 AM
    Does anyone know how long it takes for a new Prefect release to appear on conda? I can install 2.7.0 with pip but not with conda - I keep getting a package not found error.
    👀 1
    e
    • 2
    • 2
  • t

    Tibs

    12/04/2022, 9:43 AM
    Hi, we are running prefect 2.6.5 on ECS, until Friday, everything was working fine, now flows fail with: State message: Submission failed. TypeError: Infrastructure.prepare_for_flow_run() got an unexpected keyword argument 'deployment', is this a known issue?
    a
    • 2
    • 2
  • g

    Gio

    12/20/2022, 4:52 PM
    Hi everyone! I am just getting started with Prefect 2 - I'm excited to get started, but please let me know if this is not the right place for this question I am trying to include a database transaction in a flow that I am creating - so the the database will be updated only if all SQL operations are successful. Basically what is described here in the Postgres documentation: https://www.postgresql.org/docs/current/tutorial-transactions.html Is this possible to do with
    prefect_sqlalchemy
    ? I have not been able to successfully create a transaction using it.
    ✅ 1
    a
    • 2
    • 2
  • s

    Stephen Herron

    01/04/2023, 2:09 PM
    does a prefect-gmail integration exist?
    ✅ 1
    j
    • 2
    • 3
  • t

    Tibs

    01/05/2023, 1:16 PM
    Hello, for prefect_sqlalchemy integrations, is it possible to execute multiple statements queries?
    a
    • 2
    • 14
  • t

    Tibs

    01/10/2023, 7:17 AM
    Hi, I get a warning when using prefect-aws components to build blocks:
    Warning!  Failed to load collection 'prefect_aws_ecs': ImportError: cannot import name 'AwsCredentials' from partially initialized module 'prefect_aws' (most likely due to a circular import)
    My import statement:
    from prefect_aws.credentials import AwsCredentials
    from prefect_aws.s3 import S3Bucket
    from prefect_aws.ecs import ECSTask
    prefect-aws version 0.1.8
    a
    • 2
    • 11
  • d

    Dylan Lim

    01/17/2023, 10:10 AM
    Hi there, I am using
    prefect-sqlalchemy
    and have set up a
    SqlAlchemyConnector
    block. However, I notice that there is an ‘Optional’ fetch size that defaults to 1 and impacts my queries eg.
    fetch_many
    . Is there a way to prevent this fetch size from being enforced, especially if I don’t know how many rows I am retrieving beforehand?
    a
    • 2
    • 3
  • t

    Tibs

    02/01/2023, 2:17 PM
    Hi, I am currently running prefect flows using ECSTask infrastructure block setup, in an ECS Fargate cluster, it takes around 3-4 minutes for a flow to start, does anyone have some advice on how to speed up the process?
    c
    j
    n
    • 4
    • 7
  • m

    Meng Si

    02/14/2023, 8:58 PM
    Hi Prefect team, my company is migrating from Prefect1 to Prefect2. We have a few Prefect1 flows with Great Expectation integrations. But I haven’t found much information on how GE works with Prefect2. Could someone please share some information? Or is it just the same as in Prefect1?
    ✅ 1
    r
    • 2
    • 2
Powered by Linen
Title
m

Meng Si

02/14/2023, 8:58 PM
Hi Prefect team, my company is migrating from Prefect1 to Prefect2. We have a few Prefect1 flows with Great Expectation integrations. But I haven’t found much information on how GE works with Prefect2. Could someone please share some information? Or is it just the same as in Prefect1?
✅ 1
r

Ryan Peden

02/14/2023, 9:30 PM
It's not exactly the same, but you can use
prefect-great-expectations
to run Great Expectations validations: https://prefecthq.github.io/prefect-great-expectations/ See here for a look at the parameters you can pass to the run_checkpoint_validation Task. I don't think we have a longer example available right now, but keep in mind Prefect 2 tasks are often simpler to work with than Prefect 1 tasks. With Prefect 1, you needed to first create an instance of the
RunGreatExpectationsValidation
task before calling it in a flow. With Prefect 2, you just install the
prefect-great-expectations
collection, import the
run_checkpoint_validation
, and run it like you'd run any other Python function inside your flow.
m

Meng Si

02/14/2023, 9:51 PM
Great, thanks Ryan!
👍 1
View count: 2