Hi when I am raising a FAIL signal the logger logs with the INFO level. it is possible to get it to log with the ERROR level instead?

raise signals.FAIL(message=f"Task failed") # This should log with ERROR level, not INFO

Hy I am trying to return list of object from aws s3 bucket. I am getting this error when using it in prefect flow “TypeError: can’t pickle _thread.lock “objects using s3_client.list_objects(bucket,prefix) Otherwise its returning correct output without prefect on jupyter notebook

Hi everyone! I encountered an error when attempting to restart a flow from the failed task and was wondering if anyone has some best practices on how to handle this? Context: • I have a "Flow of Flows" that triggers a few connectors from Fivetran in parallel then triggers a dbt job run for the data modelling for the source once the data has finished syncing • There is one dbt job that is then run after all the prior dbt jobs have succeeded (this is the one that failed in this instance) • After fixing the error, when I press restart I get an error saying ◦ This then returns a list of all the flow runs on this flow id Question: • What would I need to specify in the task parameter so that it only re-runs the latest flow when I press restart on the UI? Thank you so much in advance for your time and help! 😄

Hi! I have a project that started with flow.py, now I need to create another flow, is it ok that I just create the new flow (db-flow.py) in this same folder?

Hello, I'm trying to trigger a flow using an AWS S3 PUT event - through Lambda function. Whenever there is an event, flow get scheduled but is stuck there. I have gone through other messages in the community - but there is no problem with labels as well. Can anyone help me with this? I used prefect cloud. Thank you.

im getting dask timeouts with my temporary clusters. im running dask in gke with prefect cloud, and im seeing a flow hang on

Creating a new Dask cluster with `__main__.get_executor.<locals>.<lambda>`...`

in my gke logs, im seeing

raise TimeoutError( asyncio.exceptions.TimeoutError: Nanny failed to start in 60 seconds

When calling

future.get_state()

, a 403/timeout response will cause flows to fail - is there a way to see api limits for prefect orion cloud? We get forbidden (403) occasionally - it appears to be due to throttling

Hey, I’m trying to configure Prefect’s logger and add a Logz.io handler. Can’t find any good example - can anyone help me with this? I configured a logger using this documentation https://docs.logz.io/shipping/log-sources/python.html but I can’t make Prefect’s “prefect.context.get(logger_name)” use it. Thanks.

hello, are artifacts stored in the postgres db? or somewhere else?

Hi, I have a flow that contains a while True loop and it starts to run on a schedule. However, I also want it to stop at some point. How should I do that? Is there a way to kill running flow runs? I'm talking about 2.0. Thanks!

Hello, I have created a project on prefect cloud, a simple hello world flow using the getting started documentation, a pending flow-run and installed a kubernetes agent on a GCP Kubernetes cluster The kubernetes agent is started and authenticated to prefect, however it doesn't pickup any pending flows. It is just stuck on "Waiting for flow runs... " Any idea what I am missing ?

Hi there, I had a few small questions about Orion: - Are there plans to continue to support module/git/bitbucket as storage options in Orion? - Is it possible to specify parameters for a flow run through the UI in Orion? - In the Prefect 1.0 docs, it's mentioned that Hashicorp Vault is used to back secrets, are there plans in Orion to allow access to our own Vault instance?

NEWBIE Here:

Hi, Is there a possibility of non-linearity and cross- and intra-flow/ work flow interaction in

prefect 2.0

? Need to do some POC.

Hello, I am new to prefect and investigating several use-cases to assess if Prefect would be a good fit in the toolbox to build a team of engineers dealing with data integration between our company's API and our customers systems. For some use-cases we have to periodically pull files from a remote SFTP server, do some manipulations with the file and some APIs then finally upload a sort of "response" file over the SFTP server. I couldn't find something related to SFTP in the built-in tasks list https://docs.prefect.io/api/latest/tasks/function.html What would be the best way to deal with SFTP file exchange with prefect ? Should I just use a custom function task and a python sftp library or is there a more suitable task ?

Can someone help with a flow running problem using Github storage?

I'm a bit confused by the documentation, why are there two documentations with both having similar sections ? https://docs.prefect.io/ "Core Workflow Engine" and "Orchestration & API" It seems to have redundant pages between them but also different pages

Hey, is there a way to add a sort of a callback function when a task fails ?

I have a question if anyone has a second. Our group is starting to port a lot of airflow DAGs to prefect, and we’ve been using some patterns that include defining a storage option when a Flow is defined. I have just worked up a new Flow (my first one) and I started from scratch. I have not had to configure or use storage, and I don’t really think I understand what the role of storage is in a flow’s definition. I’ve read the linked docs entry, and I still am confused. Can anyone explain what storage is at a high level? Thank you!

Hey there ! Got a little logger issue. I'm using a custom library in a task that uses

from loguru import logger

and uses that for logging. Unfortunately, those don't appear in Prefect logs which is not great. I've tried adding

PREFECT__LOGGING__EXTRA_LOGGERS="['loguru']"

in my environment but doesn't seem to work. Is there another way to do this ?

i have 4 simple flows i ran periodically every hour and for some reason it looks like i am running more than 20k tasks, are clock registrations being multiplied in some way?

Anyone knows how to contact prefect support?

I can't seem to get the task_run_name callable to work with mapped tasks. I'm using a method near identical to the example: https://docs.prefect.io/core/concepts/templating.html#callable-name-generators, but running the mapped task uses the method name instead of the string that's returned?

Hi all! Is there a way to disable task checkpointing at

Flow

initialization? Or is the best way to unset

prefect.config.flows.checkpointing

before building the

Flow

?

In prefect 1.X what is the best way to execute a task only if the previous data retrieval task returned results? Like call an email task only if there is data to email.

I’ve got an issue with prefect v1 and the ECS agent: I’ve got a flow that’s spinning on

Submitted for execution: Task arn:<task-arn>

with no further logs generated. In cloudwatch, it only shows:

2022-05-17T16:18:52.575-04:00	[2022-05-17 20:18:52,575] INFO - agent | Deploying flow run f2f45aac-ccfb-4bb9-88db-fb1d00426989 to execution environment...
	2022-05-17T16:18:53.805-04:00	[2022-05-17 20:18:53,805] INFO - agent | Completed deployment of flow run f2f45aac-ccfb-4bb9-88db-fb1d00426989

Is this the right way to specify a specific url for a specific internal package? This is a part of the flow.run.config

env={"EXTRA_PIP_PACKAGES": "mypackage--index-url=<https://my_url>"})

Hi all, I have a flow run that I cancelled which still has a task that appears to be running for over 3 hours now, is there any way to force kill this through the UI?

hi community! i'm trying to run a flow on AWS ECS using the ECS run config (https://docs.prefect.io/api/latest/run_configs.html#ecsrun) but my flow get stuck in a submitted state (already checked-out https://discourse.prefect.io/t/why-is-my-flow-stuck-in-a-submitted-state/201). I can see in the AWS console that the ECS task fails because the run command is

Command ["/bin/sh","-c","prefect execute flow-run"]

(however i need to run prefect in bash with a conda environment activated). Is there any way to configure the ECS task to remove the

["/bin/sh/", "-c",]

before the prefect command? My container is configured with the appropriate entrypoint, runs bash and activates conda at startup, but

/bin/sh -c

undoes that setup

anyone seen this kind of error message before:

prefect.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': 'State update failed for task run ID d253a979-2ad4-4add-9c60-e569d75758e6: provided a running state but associated flow run 165e6a96-0e0d-464b-9a85-fc97454b9375 is not in a running state.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

I have a flow that kicks off a few hundred flow runs, and it was working fine in a test environment, but I got this as soon as I tried to run it in a production environment