@Stephen Herron has joined the channel

@redsquare has joined the channel

@Claire Herdeman has joined the channel

@Nate has joined the channel

@John Mizerany has joined the channel

Hi all! 👋 Friendly neighborhood PMM here; thought I'd kick this channel off with a quick poll/survey!

Which of the following tools do you use most often with your current data stack?

🎉 prefect-gitlab integration just released! marvin This includes 2 blocks for GitLab credentials and GitLab Repository. https://github.com/PrefectHQ/prefect-gitlab

📣 Say hello to *prefect-discord*~! Brought to you by our very own @Ryan Peden P ! "A collection containing a notification block making it easy to send messages and notifications from Prefect to a Discord channel." 👾 https://github.com/rpeden/prefect-discord

hey there. Is there any plan for a

openlineage

integration? For ref: • https://openlineage.io/ • https://github.com/OpenLineage/OpenLineage/issues/81 • there was one for Prefect v1 https://github.com/OpenLineage/OpenLineage/pull/293 CC @limx0 @Anna Geller

Hi everyone, we are migrating from prefect 1 to prefect 2, in our previous flows we are using sqlalchemy ORM to run queries on our database, would you recommend to switch to the prefect-sqlalchemy library and use plain SQL? (edite

Hi everyone, is there a way to specify secrets for an ECSTask block? Something similar to what can be done in the task definition? (see attached image from AWS docs)

Hi, I am looking for some help with prefect_sqlalchemy for how I could customize the sqlalchemy_execute task? Like giving a custom name, number of retries if it is possible

Hi all. We use conda to create virtual environments for our Python code, and would like to do the same for running our flows. I noticed that the base

prefect

package is available on

conda-forge

but the integration packages are not. I raised https://github.com/conda-forge/prefect-feedstock/issues/179 about this but I wasn't sure if that was the right place for it so it would get noticed. Is there any guidance on whether you can make the Prefect-maintained integration packages available via conda as well as pip?

Hi everyone, we are having some issues with prefect on AWS. We use an S3 block to store (so using the s3fs), the we use boto3 in some of our tasks. The problem is, s3fs depends on an older botocore then our boto3 version, and we don't want to downgrade it. Does anyone have any recommendation on how to solve this? Will it be a problem in prefect_aws module?

Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This article will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection.

x-posting here with some information from Fivetran on our new Collection!

Reposting a question here for this audience: Has anybody used

prefect-airbyte

to trigger syncs on Airbyte Cloud? Or is it only for open source hosted instances?

Does anyone know how long it takes for a new Prefect release to appear on conda? I can install 2.7.0 with pip but not with conda - I keep getting a package not found error.

Hi, we are running prefect 2.6.5 on ECS, until Friday, everything was working fine, now flows fail with: State message: Submission failed. TypeError: Infrastructure.prepare_for_flow_run() got an unexpected keyword argument 'deployment', is this a known issue?

Hi everyone! I am just getting started with Prefect 2 - I'm excited to get started, but please let me know if this is not the right place for this question I am trying to include a database transaction in a flow that I am creating - so the the database will be updated only if all SQL operations are successful. Basically what is described here in the Postgres documentation: https://www.postgresql.org/docs/current/tutorial-transactions.html Is this possible to do with

prefect_sqlalchemy

? I have not been able to successfully create a transaction using it.

does a prefect-gmail integration exist?

Hello, for prefect_sqlalchemy integration, is it possible to execute multiple statements queries?

Hi, I get a warning when using prefect-aws components to build blocks:

Warning!  Failed to load collection 'prefect_aws_ecs': ImportError: cannot import name 'AwsCredentials' from partially initialized module 'prefect_aws' (most likely due to a circular import)

My import statement:

from prefect_aws.credentials import AwsCredentials
from prefect_aws.s3 import S3Bucket
from prefect_aws.ecs import ECSTask

prefect-aws version 0.1.8

Hi there, I am using

prefect-sqlalchemy

and have set up a

SqlAlchemyConnector

block. However, I notice that there is an ‘Optional’ fetch size that defaults to 1 and impacts my queries eg.

fetch_many

. Is there a way to prevent this fetch size from being enforced, especially if I don’t know how many rows I am retrieving beforehand?

Hi, I am currently running prefect flows using ECSTask infrastructure block setup, in an ECS Fargate cluster, it takes around 3-4 minutes for a flow to start, does anyone have some advice on how to speed up the process?

Hi Prefect team, my company is migrating from Prefect1 to Prefect2. We have a few Prefect1 flows with Great Expectation integrations. But I haven’t found much information on how GE works with Prefect2. Could someone please share some information? Or is it just the same as in Prefect1?

Hi! I am trying to solve an issue regarding ECSTask: using a pre-defined task definition, but still customize it with parameters given in the ECSTask, without creating a new family.

I have noticed that running

cursor.execute

with the SqlAlchemy block doesn't persist inserts. The block can connect to the db just fine and even fetch records just fine. But it can neither do inserts nor updates. FWIW, I'm executing from a jinja template like so:

cursor.execute(insert_records.render())

Hi, I am running a deployment multiple times from a master flow with different parameters.. On Sunday, 2 of the child runs failed with error

Submission failed. IndexError: list index out of range

, does anyone know what may be the cause of this? Looks like an error in prefect-aws - traceback from prefect agent in thread prefect-aws version 0.2.4 prefect version 2.8.2 flows run using ECSTask