Hi, is there a way to safely delete Prefect logs from Postgres without messing up scheduled flows? The tables log, task_run and task_run_state can grow rapidly in size, so I want to delete them every X days, but only for the ones that are ran and not scheduled.

Are there are examples of running the

prefect build/register

commands in CircleCI, specifically done in parallel? I have dozens for flows to register via CircleCI, and I'd like to do then in parallel to minimize the time.

Hi everyone, I'm trying to configure user access to the prefect, is it possible to configure using prefect outside the cloud?

when I open the home page the members button is not enabled

Hi Prefect community! Does anybody have any examples of

big_future = client.scatter(big_dask_dataframe)

and passing

future = client.submit(func, big_future)

as an output from one task to be used as an input in another task? I found this UserWarning at the bottom of the "prefect-etl" article in the dask docs (https://examples.dask.org/applications/prefect-etl.html) as well. Was wondering if anybody has encountered this issue as well? And whether there's a solution to this. Thank you in advance!

Greetings! Currently Learning about Prefect at Microsoft Build! Excited to be here!

Hi Prefect community! Just found out about you via MS Build!

Here from MachDatum from MSBuild

Does anyone have, or know of, a GitLab CI/CD image that is setup for prefect?

Hi Guys, I am trying to use Github Storage, but run into the following error:

Failed to load and execute Flow's environment: BadCredentialsException(401

Any advice how to proceed to solve this error?

Hi, right now I am storing individual flows in gitlab, and occasionally I get the error:

Failed to load and execute Flow's environment

followed by a GitLab MaxRetryError, probably because we are reading from gitlab too many times in a given time period. Using gitlab as storage is important for some of our current projects, but is there a way to retry a flow when the Flow's environment fails to load?

someone, know why when i tried to run "prefect register --project "my proj" -p /home/user/app_flows/” in a .sh file (to auto register my flows when my VM turns up) don’t work? But running the same line in the terminal works... The triggered error is prefect.utilities.exceptions.AuthorizationError: [{'path': ['project'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]

I'm running into some issues with the built in Snowflake task. Is there documentation on how to use it? Specifically I'm running into the following issue: I'm using the task SnowflakeQuery to execute a query. I'm doing that like so:

from prefect.tasks.snowflake.snowflake import SnowflakeQuery

sf_task = SnowflakeQuery(query=SNOWFLAKE_QUERY,
               user=SNOWFLAKE_USER,
               password=SNOWFLAKE_PASSWORD,
               account=SNOWFLAKE_ACCOUNT,
               warehouse=SNOWFLAKE_ACCOUNT,
               database=SNOWFLAKE_DATABASE,
               role=SNOWFLAKE_ROLE,
               schema=SNOWFLAKE_SCHEMA)
ret = sf_task.run()

Going off the source code I would expect that to return the query data in an iterable. This based off the code within

SnowflakeQuery

that creates a cursor, connects, executes the query and then returns the

.fetchall()

method of an executed Snowflake cursor (basing my understanding of that from this) What is actually returned by

sf_task.run()

is the Snowflake cursor which appears to have already executed the query (it's of type

snowflake.connector.cursor.SnowflakeCursor

with a closed connection state). However, there is metadata in

ret.description

which is a list of tuples. I've tried this with a few different queries that should return data and it's the same result. Any idea what's going on? I may be doing something obviously wrong but not seeing it. I'm able to get the data using the Snowflake connector and executing myself:

import snowflake.connector
import pandas as pd

#create connection
conn=snowflake.connector.connect(
               user=SNOWFLAKE_USER,
               password=SNOWFLAKE_PASSWORD,
               account=SNOWFLAKE_ACCOUNT,
               warehouse=SNOWFLAKE_ACCOUNT,
               database=SNOWFLAKE_DATABASE,
               role=SNOWFLAKE_ROLE,
               schema=SNOWFLAKE_SCHEMA)

                
#create cursor
curs=conn.cursor()

#execute SQL statement
cur = curs.execute(SNOWFLAKE_QUERY)

#load it to df
df = pd.DataFrame.from_records(iter(cur), columns=[x[0] for x in cur.description])
df

Can the interactive GQL API query flow-runs that are older than two weeks old?

I have created a single flow that I want to run against different environments at the same time. For example I have dev, preprod, and prod environments. Based on the given environment I can pass in the the given parameters that have the correct values(usernames, password, etc.). How do I do this with only one flow ... how do I register that flow with different parameters? Is this possible.

How do I remove AgentConfigs created in the Automation section in Cloud? I have a hard time figuring out which of my configs correspond to which config ID, because there are extra configs created called “Unnamed config” which gets selected after saving.

Hello all!  I am currently investigating whether Prefect would fit our use case. We have many machine learning tasks that are chained together in various flows. The flows are triggered when a user uploads a file on the web client and run in the background. We are considering using Prefect to schedule and manage these flows. Currently, we have many workers for each of the machine learning tasks. There is a service that schedules the jobs and processes the outputs of the jobs via a message queue. However, there is no definition of a flow and the scheduling logic is scattered throughout the code for this service. This is where we think Prefect would be useful for us. From what I gathered, Prefect is mainly used for ETL workflows. In our case, we want to use Prefect to serve results from our machine learning models. Would Prefect be suitable for our use case? I am quite new to the MLOps field so any help or suggestions would be welcome!

Hi everyone — I’m trying to use the prefect S3List task but I’m struggling with the AWS authentication, since it appears the S3List task doesn’t accept boto_kwargs nor a boto_session, and I need to specify different profiles for my task. Has anyone faced this problem before?

hey everyone. i’m suddenly starting to get this error on my flows:

AttributeError: 'Flow' object has no attribute 'terminal_state_handler'

any idea what could be the reason? i’m running on prefect self hosted server

Hi everyone — I have a task that is mapped over a list, and I have a subsequent task that must only run after all the branches of the previous task map are done, however the tasks are not data dependant. I’m trying to specify “upstream_tasks” but it’s giving me a weird error as it appears it is trying to run the task?

Is there any way to get the full traceback with a state handler?

Hi - we have a flow that occasionally gets stuck in a running state until we manually cancel it. We tried using the automations feature, but we encountered this situation last night and it didn't work

Join my webinar today with LabelStudio!

Hello Prefect! Can somebody provide some guidance on the best way to test failure paths in Prefect flows? Can I inject state into a flow during setup in someway to ensure the failure behavior is working as expected? Thanks in advance!

hi guys. I have a question regarding management of Prefect flow schedulers via Prefect API. We decided to use Prefect to handle uploading user data into our db. we support multiple different ways to upload the data (each of them is a separate Prefect Flow). In our system, we allow user to schedule a periodic job that downloads user data from external source and uploads them into our database. User should be able to specify different run periodicity for different datasets (and modify this scheduling interval). Since schedulers are stored inside a JSONB column attached to flow group without a specific schedule_id, it is very hard to update a specific schedule (unless you use one Flow parameter as a unique identifier - this feels a bit hacky). Also removing a schedule from the flow group is slightly uncomfortable (Prefect UI simply sends the whole schedule array without the deleted schedule back to graphql). At the moment, we are exploring the possibility of registering Prefect flows per dataset (using dataset ID in flow name) so we can reliably control the scheduler. This should give us better control over the scheduler, but we also get more granular view of the flows (which we dont really need or want). Is there a better way of approaching this particular problem? Any suggestions are welcomed.

Hi everyone — I’m trying add stuff to the prefect.context inside a task, to be used in other tasks, but it seems like the context is reset between tasks?

Hi, question regarding this guide: https://medium.com/the-prefect-blog/seamless-move-from-local-to-aws-kubernetes-cluster-with-prefect-f263a4573c56 . If I want to have a prefect agent running in my EKS cluster, do I need to allow any inbound traffic, or is it enough to just allow the cluster to access the public internet? Also, is there a more restrictive rule I can have on the outbound end so I don't need to allow all traffic?

Hello, every time I start the flow, the prefect does not find the path where my scripts are and the same error is displayed Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'download'")

Hi there, I'm new to Prefect and checking out options to see if Prefect will work for my company. I'm having trouble starting an ECS Agent within our infrastructure that communicates with PrefectCloud. We are using HIPAA compliant AWS architecture as described here, i.e. most of the infrastructure is in private VPCs and exposed via a public VPC. Can you help me figure out if that's my issue?

Hi everyone — on one of the first tasks in a flow I create an aws session object that is then passed as argument to many subsequent tasks. Is there any way to set this aws session globally for all tasks, so that I don’t have to explicitly pass it? I tried adding it to prefect.context but that doesn’t work..