Our Prefect Cloud 1.0 runs are piling up, and not executing. Is there a Prefect outage?

Heya! We occasionally receive fake-alerts such as flow has been pending for > 15 minutes, while it actually completes in ~30 seconds 🤷 In the logs of the flow, I can see it started at 115958 PM and completed successfully at 120031 PM (UTC). The Cloud's automation trigger is set to Stays in Pending for 15 minutes, but the alert comes in around the time the flow is done (~30 seconds). Also noticed just now that the

Stays in

now is called

Stays in (beta)

.. this is new! Are we not to rely on this automation, because it is "beta" and is broken/incomplete?

hey, would like to follow on, https://github.com/PrefectHQ/prefect/issues/7742 I am getting with Prefect Cloud and Windows Agent Any idea?

Hey all, I'm wondering if there is a known issue with Automations, specifically sending notifications for flows entering failed state? I'm fairly certain it's broken right now.

Hi, I am trying to push following as config ''' return Deployment.build_from_flow( flow=func, name=self.schedule_config.name, work_pool_name=self.schedule_config.work_pool_name, parameters=self.schedule_config.input_params, version=self.schedule_config.version, schedule=( CronSchedule( cron=self.schedule_config.cron_expression, timezone=self.schedule_config.time_zone, ) ), is_schedule_active=self.schedule_config.schedule_enabled, )

it does not support and throwing expression - TypeError: Type is not JSON serializable: ListConfig

Does 2 node array supported as part of config, it is being used to pass multi-run pass on hydra

Hi, is anybody else having problems creating prefect deployment from a python venv ? I am getting an import error from a prefect flow when I create a prefect deployment from a venv.

Hi there. I want to use GCP Google Compute Engine as prefect agent and run a flow through prefect cloud api. I have deployed my code to a gcs bucket. My code is all working fine locally. However, when I run prefect agent start -p mypool on the VM, the flow fails as follows.

14:14:30.142 | ERROR   | prefect.infrastructure.process - Process 'psi6-vanden-sector' exited with status code: -9; This indicates that the process exited due to a SIGKILL signal. Typically, this is either caused by manual cancellation or high memory usage causing the operating system to terminate the process.

The above error is not a memory issue (memory did not spike when I monitored it), nor did I manually terminate it. One peculiarity is that retrying this failed flow results in a new error message:

prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.
14:22:03.283 | ERROR   | Flow run 'psi6-vanden-sector' - Finished in state Failed('Flow run encountered an exception. MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.\n')

Hi everyone. I’ve set up an email notification block. Unfortunately, the domain of one of the email addresses to notify have had some dns issues, hence the email has been bouncing any notifications send using this block. Now dns issues have been fixed, however it doesn’t receive any notifications. I assume the email has been blocked somewhere (at least that’s my experience from Amazon SES). Anyone who experienced similar issue and know how to resolve it?

I have a flow that ran fine from cloud 24 hours ago. This morning (at 5am!) the flow run says “Downloading deployment from None” followed by a file not found for the flow’s python file in my deployment.

Hi! I am facing some issues in our deployment after the owner changed in our instance.. Right now I'm seeing this error message from the agent in the compute engine in GCP: "httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://api.prefect.cloud/api/accounts/8cba8c22-3359-43ea-845e-b88737a33ddd/workspaces/caa7e1bf-faa2-4ba5-baab-1987efa226f1/work_queues/name/prod' For more information check: https://httpstatuses.com/401 An exception occurred." Getting this when checking the /health endpoint (see image) However, I've updated the PREFECT_API_URL and the PREFECT_API_KEY. Locally flows run fine. Not sure what's causing this and how to fix it.. would anyone be able to point me in the right direction?

Anyone seeing an increase of

prefect.exceptions.PrefectHTTPStatusError: Server error '502 Bad Gateway'

after the previous Cloud release/maintenance window?

Hi everyone! When migrating from Prefect1 to Prefect2 I encountered a problem. Basically I call methods in Tasks and in those methods, I log my outputs to local .txt files. When I run the flow with Prefect1 cloud, it works perfectly fine. But when I migrated to Prefect2 cloud, It doesn't write logs to my local .txt files. I tried so many things but none of them worked. Is there any possible solutions? Thanks a lot!

This morning I had a flow run fail with the following error. This happened after all the flow code itself had completed.

04:00:42  Completed successfully
04:00:42  
04:00:42  Done. PASS=9 WARN=0 ERROR=0 SKIP=0 TOTAL=9
07:00:42 AM dbt seed --full-refresh-2 prefect.task_runs

07:01:19 AM
prefect.flow_runs
Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '502 Bad Gateway' for url '<https://api.prefect.cloud/api/accounts/f11f03de-8f9c-4660-bbba-f00d85087da4/workspaces/00985c36-8720-46ae-a667-258da45f0916/task_runs/691bdd4f-ff86-48c2-bd30-9ecad8bffe86/set_state>'
For more information check: <https://httpstatuses.com/502>

It's not the first time this has occurred. Any ideas what happened?

I use Prefect Cloud 1 and have a Linode VM to run the Python scripts, it works great (yes, I need to upgrade quickly)!!!

For a new client I'm wanting to use Prefect Cloud 2 - is there a way to host the Python scripts on github ? or do I still need to use the VM ? (and this is for a completely new client, no previous Prefect knowledge)

We have an automation set up to alert us, when a queue becomes unhealthy. I've checked the prefect agent logs that is polling the problematic queue and it has reported no errors at all 😕 Automation details Trigger Type: Work queue health Work Queues: <queue 1>, <queue 2> Work Queue: Stays in Unhealthy For: 2 Hours First

prefect-cloud.automation.action.executed

event details look like so

{
  "id": "ffcf0a09-c7cd-48f0-862a-a964474d0755",
  "account": "c1397d5f-b9f3-49e8-abb6-bce7d7b1412e",
  "event": "prefect-cloud.automation.action.executed",
  "occurred": "2023-03-20T19:19:49.450Z",
  "payload": {},
  "received": "2023-03-20T19:19:49.451Z",
  "related": [
    {
      "prefect.resource.id": "prefect-cloud.automation.82a15699-e855-4973-8c84-795b3b54ecb4",
      "prefect.resource.role": "automation",
      "prefect-cloud.name": "Production work queue unhealthy > 2h -> Slack",
      "prefect-cloud.posture": "Proactive"
    }
  ],
  "resource": {
    "prefect.resource.id": "prefect-cloud.automation.82a15699-e855-4973-8c84-795b3b54ecb4.action.0",
    "prefect-cloud.action.type": "send-notification"
  },
  "workspace": "32dfe242-315b-4405-b06d-8b6308d6b631"
}

What are the next steps to troubleshoot this? P.S. I'm running prefect agent via

systemd

on an EC2 (Ubuntu 22.04.2 LTS).

Hello. When I run a shell in python in docker using prefect, I get an error /bin/sh: 1: yc: not found. If I try to run script in docker container it works correctly. For example:

>>> import subprocess
>>> subprocess.run('yc --help',  shell=True, capture_output=True, text=True)

Hi! Does Prefect Cloud support RBAC role changes using IaaC?

Hello! I am still running into an issue where some of my Prefect 1.0 cloud runs are stuck in a scheduled state. How can I begin triaging this? It seems that some other users were also having the same issue a few days back https://prefect-community.slack.com/archives/CM28LL405/p1678385552717259

Hi, does anybody know how to unregister a block type?

curious to know, I'm hitting an error on my long flow run (its been running around 40mins) that says that my flow was cancelled (TerminationSignal: I didn't cancel it!) wondering if it could be due to hitting an api rate limit. Would I see a 429 error within my Prefect logs if it were because of hitting an api limit on number of logs allowed? I'm on a personal free account.

Is anybody else experiencing significant delays with the Blocks page loading? Sometimes a blank page loads saying I have no Blocks and I have to refresh like 5 times to get my blocks to show up. Another user is experiencing the exact same.

this is new..used to be able to just enter my parameters as strings, didn't need to provide JSON, wish there had been a headsup or something

Is the new result-view-feature already available in cloud? I can only see a link pointing to the result-persistence documentation, but no tracked results. I have result-persistence on with

persist_result=True

,

cache_result_in_memory = False

. Result persistence seems to work, because task-runs are cached. I am using the

JSONSerializer

.

Hello, I am migrating some of my prefect v1 flows to v2. But I am having a issue with running a task from another task. I read in the documentation this can be done by calling .fn(). I am trying to do this with the prefect_gcp package on the cloud_storage_upload_blob_from_string task. When executing this from the function with a @flow decorator it works. However when calling this from a task directly with .fn() it seems to do nothing and just skip the execution. Anybody has a idea?

Hi all, I got a question from our security team. Prefect-cloud, does it contain the logs of the tasks being run? They are trying to ascertain how much corporate knowledge is being sent to Prefect Cloud.

And also, is there a guide or methodology to get prefect working on a disconnected network?

Hello! Is there a difference between running a script in flow and running a script in docker? I'm trying to get data from kubernetes and when the flow starts, it writes an error, and if you run this script in the docker, it will work.