prefect-community #prefect-integrations

Docs

Join the conversation Join Slack

prefect-integrations

Oscar Björhn

11/01/2022, 3:04 PM

@Oscar Björhn has joined the channel

Rob Freedy

11/01/2022, 3:10 PM

@Rob Freedy has joined the channel

Charleigh Liang

11/01/2022, 3:10 PM

@Charleigh Liang has joined the channel

Stephen Herron

11/01/2022, 3:11 PM

@Stephen Herron has joined the channel

redsquare

11/01/2022, 3:11 PM

@redsquare has joined the channel

Claire Herdeman

11/01/2022, 3:28 PM

@Claire Herdeman has joined the channel

Nate

11/01/2022, 3:30 PM

@Nate has joined the channel

John Mizerany

11/01/2022, 3:33 PM

@John Mizerany has joined the channel

Charleigh Liang

11/01/2022, 5:41 PM

Hi all! 👋 Friendly neighborhood PMM here; thought I'd kick this channel off with a quick poll/survey!

:prefect: 4

🚀 4

user

11/01/2022, 5:42 PM

Which of the following tools do you use most often with your current data stack?

Kalise Richmond

11/03/2022, 9:22 PM

🎉 prefect-gitlab integration just released! :marvin: This includes 2 blocks for GitLab credentials and GitLab Repository. https://github.com/PrefectHQ/prefect-gitlab

🦊 6

😀 5

👍 6

:blob-attention-gif: 4

Bianca Hoch

11/04/2022, 9:14 PM

📣 Say hello to *prefect-discord*~! Brought to you by our very own @Ryan Peden 😛refect: ! "A collection containing a notification block making it easy to send messages and notifications from Prefect to a Discord channel." 👾 https://github.com/rpeden/prefect-discord

:party-parrot: 4

:gratitude-thank-you: 2

🔥 7

🎉 3

🚀 3

🙌 3

Ricardo Gaspar

11/08/2022, 4:35 PM

hey there. Is there any plan for a

openlineage

integration? For ref: • https://openlineage.io/ • https://github.com/OpenLineage/OpenLineage/issues/81 • there was one for Prefect v1 https://github.com/OpenLineage/OpenLineage/pull/293 CC @limx0 @Anna Geller

Tibs

11/16/2022, 10:43 AM

Hi everyone, we are migrating from prefect 1 to prefect 2, in our previous flows we are using sqlalchemy ORM to run queries on our database, would you recommend to switch to the prefect-sqlalchemy library and use plain SQL? (edite

✅ 1

Tibs

11/17/2022, 1:31 PM

Hi everyone, is there a way to specify secrets for an ECSTask block? Something similar to what can be done in the task definition? (see attached image from AWS docs)

✅ 1

Tibs

11/22/2022, 4:50 PM

Hi, I am looking for some help with prefect_sqlalchemy for how I could customize the sqlalchemy_execute task? Like giving a custom name, number of retries if it is possible

James Tisato

11/27/2022, 11:28 PM

Hi all. We use conda to create virtual environments for our Python code, and would like to do the same for running our flows. I noticed that the base

prefect

package is available on

conda-forge

but the integration packages are not. I raised https://github.com/conda-forge/prefect-feedstock/issues/179 about this but I wasn't sure if that was the right place for it so it would get noticed. Is there any guidance on whether you can make the Prefect-maintained integration packages available via conda as well as pip?

Tibs

11/28/2022, 4:19 PM

Hi everyone, we are having some issues with prefect on AWS. We use an S3 block to store (so using the s3fs), the we use boto3 in some of our tasks. The problem is, s3fs depends on an older botocore then our boto3 version, and we don't want to downgrade it. Does anyone have any recommendation on how to solve this? Will it be a problem in prefect_aws module?

✅ 1

Khuyen Tran

11/29/2022, 4:19 PM

Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This article will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection.

🙌 3

:party-parrot: 1

:gratitude-thank-you: 1

Chris Reuter

11/29/2022, 11:21 PM

x-posting here with some information from Fivetran on our new Collection!

:gratitude-thank-you: 3

🤠 3

🎉 4

🙌 3

:fivetran: 1

Michael Kahan

11/30/2022, 10:04 PM

Reposting a question here for this audience: Has anybody used

prefect-airbyte

to trigger syncs on Airbyte Cloud? Or is it only for open source hosted instances?

James Tisato

12/03/2022, 2:02 AM

Does anyone know how long it takes for a new Prefect release to appear on conda? I can install 2.7.0 with pip but not with conda - I keep getting a package not found error.

👀 1

Tibs

12/04/2022, 9:43 AM

Hi, we are running prefect 2.6.5 on ECS, until Friday, everything was working fine, now flows fail with: State message: Submission failed. TypeError: Infrastructure.prepare_for_flow_run() got an unexpected keyword argument 'deployment', is this a known issue?

Gio

12/20/2022, 4:52 PM

Hi everyone! I am just getting started with Prefect 2 - I'm excited to get started, but please let me know if this is not the right place for this question I am trying to include a database transaction in a flow that I am creating - so the the database will be updated only if all SQL operations are successful. Basically what is described here in the Postgres documentation: https://www.postgresql.org/docs/current/tutorial-transactions.html Is this possible to do with

prefect_sqlalchemy

? I have not been able to successfully create a transaction using it.

✅ 1

Stephen Herron

01/04/2023, 2:09 PM

does a prefect-gmail integration exist?

✅ 1

Tibs

01/05/2023, 1:16 PM

Hello, for prefect_sqlalchemy integrations, is it possible to execute multiple statements queries?

Tibs

01/10/2023, 7:17 AM

Hi, I get a warning when using prefect-aws components to build blocks:

Warning!  Failed to load collection 'prefect_aws_ecs': ImportError: cannot import name 'AwsCredentials' from partially initialized module 'prefect_aws' (most likely due to a circular import)

My import statement:

from prefect_aws.credentials import AwsCredentials
from prefect_aws.s3 import S3Bucket
from prefect_aws.ecs import ECSTask

prefect-aws version 0.1.8

Dylan Lim

01/17/2023, 10:10 AM

Hi there, I am using

prefect-sqlalchemy

and have set up a

SqlAlchemyConnector

block. However, I notice that there is an ‘Optional’ fetch size that defaults to 1 and impacts my queries eg.

fetch_many

. Is there a way to prevent this fetch size from being enforced, especially if I don’t know how many rows I am retrieving beforehand?

Tibs

02/01/2023, 2:17 PM

Hi, I am currently running prefect flows using ECSTask infrastructure block setup, in an ECS Fargate cluster, it takes around 3-4 minutes for a flow to start, does anyone have some advice on how to speed up the process?

Meng Si

02/14/2023, 8:58 PM

Hi Prefect team, my company is migrating from Prefect1 to Prefect2. We have a few Prefect1 flows with Great Expectation integrations. But I haven’t found much information on how GE works with Prefect2. Could someone please share some information? Or is it just the same as in Prefect1?

✅ 1

Title

Meng Si

02/14/2023, 8:58 PM

✅ 1

Ryan Peden

02/14/2023, 9:30 PM

It's not exactly the same, but you can use

prefect-great-expectations

to run Great Expectations validations: https://prefecthq.github.io/prefect-great-expectations/ See here for a look at the parameters you can pass to the run_checkpoint_validation Task. I don't think we have a longer example available right now, but keep in mind Prefect 2 tasks are often simpler to work with than Prefect 1 tasks. With Prefect 1, you needed to first create an instance of the

RunGreatExpectationsValidation

task before calling it in a flow. With Prefect 2, you just install the

prefect-great-expectations

collection, import the

run_checkpoint_validation

, and run it like you'd run any other Python function inside your flow.

Meng Si

02/14/2023, 9:51 PM

Great, thanks Ryan!

👍 1

View count: 2