@Marvin, when I try to change my API key to a custom one that never expires, I get the following error when I try to login to the key in my terminal window "Cannot log in with a key when a different PREFECT_API_KEY is present as an environment variable that will override it.". How can I fix this

Hello all. I am running into various

sqlalchemy

errors when running a test flow with many tasks (>10000) locally. Here's the code I'm running: https://gist.github.com/yhshin11/1832bc945446a62c5c6152abb9c1a0a5 It seems like the problem has to do with the fact that there are too many tasks that are trying to write to the Orion database at the same time. I tried switching to a Postgres database as described in the [docs](https://docs.prefect.io/concepts/database/), and also adding concurrency limit of 10. Neither seems to fix the issues. Any ideas about how to fix this? Here's an example of the kind of errors I'm getting:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/14/3o7r>)

Hi, all of our flow runs have been failing for the last 12 hours with the error

Submission failed. RuntimeError: Cannot put items in a stopped service instance.

Does anyone know what might be causing this? cc @Marvin

Hello 🙂 I believe I ran into a threading problem yesterday. I am converting some data extraction pipelines to prefect 2 flows, which works fine for most of them. I just add the decorators and some logging. But one of the data pipelines uses sqlite for intermediate storage, and this is the error message I receive:

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140193261803264 and this is thread id 140192735962880.

I can also paste the part of the code where sqlite is called. I am also trying to guess the reason behind the error. There is an sqlite object created in a function. Within the function I define another function which uses this object. But both python functions are not prefect flows or tasks. They live within a bigger prefect flow. So here comes my first question: Does prefect 2 create a different thread for each nested function inside a flow or a task. Otherwise I cannot explain why the 2 parts of the code would run in different threads.

is it possible to set a context variable on the flow level? for context, i want to set a flow owner to the flow and then use that value in my slack state handler. but even when i set

with prefect.context(dict(flow_owner="test")):

before defining the flow, i still get an error

Traceback (most recent call last):
  File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/lanadann/.pyenv/versions/data-prefect-3.9.7/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/lanadann/prefect/data_prefect/lib/notifiers.py", line 13, in post_to_slack_on_failure
    f"@{prefect.context.flow_owner} "
AttributeError: 'Context' object has no attribute 'flow_owner'

Hi all, what's the best practice around using RDS postgres database authentication for Prefect Agents on AWS? Do folks use IAM database authentication or Postgres Native Authentication?

hi everyone! im facing a weird issue in flow runs . im running prefect on aws eks ( self hosted instance of orion and kubernetes jobs on different namespaces ) 1. i have 9 running flows which i can see on the ui but i cant find the corresponding job on my cluster 2. when i check the concurrency-limit for the tags that these flow use i can see 14 task runs ( these flow have parallel task runs so some of them are having more than one task stuck in running state) so basically 9 running flows with 14 running tasks as i can see on the ui but i cant find the jobs on my cluster. doubt - how did these flow runs get terminated on the cluster and even if they did why did it not update in the ui or lets say why was the state not updated in meta db attaching screenshots for the same

Hello, in Orion - how to handle logging from python files which are not part of flow code? I usually use loguru for my logging but not able to make it work with Prefect logging seamlessly. I would like to have both prefect logging and app logging to sink to same target (file/console) and with consistent format. Are there any good examples available on this ?

How do I create and run flows on a schedule in Prefect Orion? Let’s say I have this:

schedule = IntervalSchedule(interval=datetime.timedelta(seconds=10))
deployment_spec = DeploymentSpec(name="hola", flow=greetings_flow, schedule=schedule)

Do I now use OrionClient and some methods to deploy? We currently use Prefect core in a pod and simply do flow.run() with a schedule attached, and I’m looking for a working example to do the equivalent in Orion (even if I gather behind the scenes it will do something else via a server and DB). Basically I’m looking for this: https://orion-docs.prefect.io/concepts/deployments/#running-deployments-with-the-api 🙂

I am working on a POC to use Prefect to help orchestrate an ELT process. I was able to get a flow and tasks to run locally and show up in the local server ui. Then i setup a cloud prefect account because i wanted to test task concurrency and got it to kick of my flow from my local machine. My next step is to see what it takes to productionize the flow in Azure, so i have a few questions. 1. I know there is a azure marketplace item to Create Prefect Agent. Is that the best way to get that part setup? Is Kubernetes the way to go over just a VM with Docker? I tried getting the kubernetes version to install but it failed on the last step with "The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant.". no idea what that means and when i tried to run the install again, it gave me "A custom role with the same name already exists in this directory. Use a different name" 2. Does anyone know of a complete walkthough of setting up an agent in Azure (VM with docker or kubernetes). i did find something, but they dont show anything on setting up the kubernetes service etc. https://infinitelambda.com/post/prefect-workflow-automation-azure-devops-aks/