Anyone seeing a bunch of prefect errors on getting secret — in the last few hours at least? Couple of folks from our end seeing it…

KeyError: 'xxx'

Hello Everyone, I have a question, but IDK if there is a channel for quick questions or something similar

@Anna Geller hi anna, is there any clear date that prefect 2.0 can support map operator? even we can write a loop to achive this , but we can't lookup the progress in that way. thank you

Hiya, is it possible to use a string concatenation of a task result as an input parameter to a downstream task? 🧵

Good Morning everyone, 👋 I have you are having a great day. My problem: I have Prefect 1.2.2 set up in an EKS Cluster . The Kubernetes agent schedules flow runs but they don't get executed only queued with a submitted state.

@Anna Geller hi Anna, I have a couple of questions relative to Prefect 2.0 - in thread.

Hi all. Ever since we started using Prefect Cloud, we have been making use of Prefect Secrets and using them across the board on almost all our flows. We have been encountering an intermittent error that happens on almost any of the flows at a random basis. Some flows don't use

PrefectSecret

explicitly but make use of secrets in other ways, such as

Git

storage

git_clone_url_secret_name=<secret_key>

. The error is the following:

ClientError([{'path': ['secret_value'], 'message': 'An unknown error occurred.', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])

The error is very vague and it's very difficult to understand what the error is exactly. Any help is appreciated, this is affecting our production workflows that sometimes take hours to run.

Hi all! I'm new here and only started with my way with Prefect. I'd like to ask if someone could give some advice or just a documentation (I haven't found any): I'm using Prefect 2.0/beta, there I'd like to create a deployment for a flow with multiple tasks, each of them should use its own docker image. For Prefect v1 I used CreateContainer from prefect.tasks.docker library, but for Prefect v2 I cannot find any replacement. Please, any advices?

I am using the

GCSResult

for my Task results. Does anybody know if there is a way to catch exceptions with it? We are seeing occasional failures on the GCS side and I don't want my flows failing because of it.

@Chris Reuter: "Can't make it? Go to our Prefect YouTube channel for the recording!" Except that the latest Fireside Chats haven't made it to the YouTube channel?

hi! i've spent hours and hours debugging my configuration with AWS ECS Fargate encountering

ClusterNotFound

but haven't gotten anywhere. i'd immensely appreciate it if anyone would have any insights! details in thread

could i check if ECS agent is ready for prefect 2 yet? Digging the updated and simplifed UI!

Prefect 2.0 Hello Everyone, Is there a way I can use multiprocessing.pool in prefect? sample code is in the thread, it works if I run it without prefect. how can I integrate it in prefect? thanks in advance.

Hello, is there an ongoing outage with Prefect Cloud? Seems some of our flows just stopped getting scheduled? I just had to manually toggle the flow off and on again to get it to produce future (scheduled) flow runs. Flow ID: f274e98a-3dcd-4823-8de7-4b74cc27615a

Hi, I have one flow where I have two steps that I use a map,

Task A

to do an upload and

Task B

to monitor if everything went fine after the upload from Task A. I noticed that using LocalDaskExecutor, only 10 tasks are processed at a time It’s possible to run more than 10 mapped tasks in parallel? Using LocalDaskExecutor with Prefect 1.2.2

Hi can someone please share a some content on handling multiple work flows.

Good day everyone, did I miss something in UI or Orion self-hosted backend is not allowing to edit parameters of deployment/flow in UI? Thanks

And another one - if flow-decorated main function is async def'ined and consists of await asyncio.gather() then FlowRunner will properly asyncio.run() it?

I think prefect 2.0 docs are off, there is line the about work-queues and agents section that states this: “You must start an agent within the environment in which it will execute flow runs.” This statement is a little off and should come with note stating the agent doesn't necessarily have to be running in the same execution environment as the flow. For instance you can have an agent running in a VM connected to a k8 cluster using kubectl and run k8 jobs that way.

Prefect 2.0 Hi, my code runs fine when I run it with python abc.py but it doesn't do anything when I run it from UI i.e prefect deployment create abc.py. it spins up a new container when I run it from UI but then shutdowns I don't see any logs for debugging, the flow status shows completed. I am using dockerflow runner with DaskTaskRunner.

Hi, I am trying to host a prefect 2.0 instance on google cloud platform compute engine. Could someone please share some documentation?

Hi I am trying to run a sync connection task in airbyte using prefect. but getting this error sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table block_schema_reference already exists. when I sync from airbyte it works.

Hi, Can you help me to do create dynamic DAG flow based on configuration file(Yaml) and also how to do parallel scheduling at the sametime @Anna Geller

Context: I have a number of tasks that are essentially API calls which can be run in parallel and take ~ 1 second each but every second of latency saved is very important. I’d like to use Prefect in production and take advantage of the native async support, but async support is only available in prefect 2.0 and that is not recommended in production. Is there any recommendations here, would it be safe to use prefect 2 in Production, or is there anyway to use asyncio in prefect 1.0 (I couldn’t find any documentation for this), or would it be a good idea to just use the DaskExecutor in prefect 1.0 with the

threads

scheduler? 🙏

morning! I have a small installation problem. Prefect requires Python 3.7+ and that's installed on our AWS Linux machines. Problem is, the default Python associated with the 'python' command is Python2. If you try to make Python3 the default incredibly you end up breaking YUM which relies on Python2. What's the best way to solve this problem? In short I need prefect to rely on Python 3 without altering the rest of my machines. I am no Python expert as you have probably guessed 🙂

Can I just say how much of a pleasure it is to work with the GQL endpoints - it seems to be all very well designed - I wonder if the code for how you setup the API is publicly available? I need to write my own and looking for insparation...

Hey community! I am currently trying to deploy a K8s Orion setup. After quite some starting problems, I thought I had finally made it. Unfortunately, now I see very weird behaviour: • The agent pod keeps restarting in a CrashLoop. The error message is very lengthy HTTP Read Timeout. Abbreviated message is

...
File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 834, in read_work_queue_by_name
...
httpx.ReadTimeout
An exception occurred.

• The agent gives these weird log messages:

MarkLateRuns took 26.306307 seconds to run, which is longer than its loop interval of 5.0 seconds.
FlowRunNotifications took 30.444981 seconds to run, which is longer than its loop interval of 4 seconds.
MarkLateRuns took 30.619028 seconds to run, which is longer than its loop interval of 5.0 seconds.

My guess is that something is really slowing the container down so that it runs into connection timeout issues as it doesn’t reply in time. Does anyone have any idea where to look further? The error message unfortunately gives me zero insights on the matter 😞 Thanks!

In prefect 2.0 the current agent solution isn't super price effective for a large number of queues. For instance if i use k8 autopilot in GCP I get charged per hour and since agents are always polling that gets pricey. I can run the agents in VM that is still expensive. Each agent running cost about 100-125 MB in memory. If i had 50 queues then i need 50 agents running which is about (50 * 150MB) we looking at a 8 GB VM to run these agents in). And double that if needed additional 50 queues. I would be looking at solution like cloud run (serverless 60 min max run time) to run the agents on. I was wondering if prefect would allow for queue watcher or alerting/webhook feature down the road?

Hi All, We have installed python 3.8, prefect 1.2.2 (2022.01 helm chart version) and dask gateway 2022.4.0 on the Google kubernetes engine cluster We have resgistered the prefect flow as well. The flow is appearing in the prefect UI.When we click "quick run", the flow run fails after couple of mins with the below error. The prefect is accessing dask gateway and running jobs. Could you please suggest.

Hi, I am trying to self host prefect 2.0 on a gcp cloud engine vm. I have installed prefect on the vm and set PREFECT_API_URL as "http//<external ip>4200/api" I tried to connect to the prefect instance from my local machine by setting the PREFECT_API_URL on my machine and running this code:

$ # start python REPL with native await functionality
$ python -m asyncio
>>> from prefect.client import get_client
>>> async with get_client() as client:
...     response = await client.hello()
...     print(response.json())
👋

and I got this error:

Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 440, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
  File "/home/shivam/.local/lib/python3.9/site-packages/prefect/client.py", line 111, in get_client
    ctx = prefect.context.get_settings_context()
  File "/home/shivam/.local/lib/python3.9/site-packages/prefect/context.py", line 272, in get_settings_context
    raise MissingContextError("No settings context found.")
prefect.exceptions.MissingContextError: No settings context found.

Am I missing something in the setup?