Woke up this morning to the Cloud's Prefect 2.8.0 update with work pools. I know that all existing work-queues went into the default work pool, however, flow runs are being tagged with either

work queue

or

work pool

in what seems to be random. I liked the old way of seeing the work queue because it gave me an idea of the execution environment of said flow. Is this intended?

Hey Prefect team,

Task run '<id>' received abort during orchestration: This run cannot transition to the RUNNING state from the RUNNING state. Task run is in RUNNING state.

prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.

I’m getting this error. I have a Dask cluster on EKS. When I run the prefect flow locally, it connects to the dask cluster and runs it just fine. When I deploy it to the cloud, my prefect agent on Fargate runs the first task in my DAG and then crashes with this error a few seconds later. I’m using Prefect 2.8.0.

where is Prefect Cloud’s maintenance schedule documented?

set the channel description: Hi 👋 - post comments & code in 🧵 - ☁️ docs: https://docs.prefect.io/ui/cloud/ - status updates: https://prefect.status.io/

Hi everyone! I need speak with someone from financial department about our plan (Logcomex). Our Contact is Richard Christie

Is there somewhere I can subscribe to upcoming change notifications for Prefect Cloud? I'm working on a v1 to v2 migration at the moment and was confused when work queues went missing (I've found them now) from the UI since the last time I looked

Hi community is there a way where I could run a deployment on prefect cloud without keeping my local PC switched on. My question is after creating a deployment and scheduling will the flow run even when my PC is disconnected.

hi there! through some wackiness, i've ended up with a couple of prefect cloud accounts based on the aliases of my e-mail address. Is there a way in prefect cloud that i can request for one of those accounts to be deleted?

Is the automations limit of 3 per workspace a free tier limit or a forever limit?

Hi, we are getting some spurious alerts from prefect-cloud: This alert is configured to trigger if a flow-run remains in

Late

or

Pending

state for over 10 minutes, but we are seeing this even if the flow completed successfully and in less than 10 minutes.

Prefect flow Late or Pending for over 10 minutes

Flow run build-meta/spectral-anteater entered state

Completed

at 2023-02-15T19:39:53.508606+00:00.

Flow run URL: https://app.prefect.cloud/account/adee12e2-25d9-428a-9a77-32d8d7ca2ac7/workspace/28562586-0812-4e92-803f-99ce64384cbd/flow-runs/flow-run/81433951-84a0-4bb1-95e9-22a942912d3a

State message: All states completed.

Since the new waterfall visualization was introduced to Prefect Cloud, we've been experiencing frequent freezing and crashing on the UI when using

map

for tasks. Is there a way to simplify the visualization so that it doesn't show every single task mapping on the UI? It also makes it rather difficult to see and understand what's happening in the flow

Heya! Last night we received quite many slack notifications of flow being healthy from single Automation over span of roughly 2 hours. All but 2 were with valid payload 😕 The amount of notifications are to be attributed to following config • Trigger Type: Work queue health • _*Work Queues*_: <single production work queue> • _*Work Queue*_: Stays in > Healthy • _*For*_: 10 Seconds Our Slack notification template looks as follows

Name: {{ work_queue.name }}
Last polled: {{ work_queue.last_polled }}
Late run count: {{ work_queue.late_runs_count }}
URL: {{ work_queue|ui_url }}

But.. why are all but 2 notifications with no values? (see screenshot below). The two valid notifications are 1. The first one at 21:53 CEST 2. At 22:54 CEST (visible in screenshot) edit: For the record.. We do not intend to run such trigger condition (it was a mis-configuration). We do not really care being notified every 10 seconds about a queue being healthy 😅 Hence the subject of the notifications in screenshot is mis-leading

Hey guys! Since yesterday we've been having issues with new runs being scheduled. We've noticed this issue in the past once or twice but not this severe. Any idea what might be happening? We are on prefect 1.0 still.

Is there anyway to restrict Prefect cloud account by API keys. Basically I am trying to find the easy way to deploy code to prod and non-prod env by API keys such that one env API key cannot access another.

👋 everyone! QQ. I am getting ready to build a little deployment that gsutil rsync's a bunch of data from s3 to gcs. Unfortunately the data in s3 doesn't really follow a nice partition hierarchy so i can't just

gsutil rsync <s3://some-key/dt=yyyy-mm-dd/>

<gs://some-key/dt=yyyy-mm-dd/>

😢 I am going to give

prefect-shell

a try for the first time and want to know if people have had a lot of experience with it? For my use case I have about 12K different rsync's I am going to need to run and I don't know which of these patterns is preferable:

for src in s3_sources_12k:
    dest = f'<gs://some-dest/{src}>'
    ShellOperation(
        commands=[f"gsutil rsync -r {src} {dest}"],
        env=env_var_map,
    ).run()

or

with ShellOperation(
    commands=[
        "gsutil rsync -r src1 dest1",
        "gsutil rsync -r src2 dest2",
        "gsutil rsync -r src3 dest3",
        ...
        "gsutil rsync -r src12k dest12k",
    ],
    env=env_var_map,
) as shell_operation:
    shell_process = shell_operation.trigger()
    shell_process.wait_for_completion()
    shell_output = shell_process.fetch_result()

Hi all! We suddenly started to have issues with flow runs on Prefect Cloud not picking up on their input parameters. Our flow runs are getting

null

for all parameters, even when there are defaults set and when we choose

Customize Run

to input parameters. Is this a known issue? This seems to have begun when the latest release of Prefect was pushed, about 3 hours ago.

I just went in to cloud for the first time in a day or two and it looks like my notification rules were migrated. They didn’t pick up a failed run from this AM. Additionally, the notification method (slack) is now sealed up in an “anonymous block” that I can’t see or edit. Do I need to recreate the automations?

Hey folks 👋 I've been playing around with custom Prefect logging. It works great when I am running flows locally, but I cannot seem to get my deployments to pick up a custom

logging.yml

set with

PREFECT_LOGGING_SETTINGS_PATH

. Does anyone have examples of how this is done for cloud deployments?

Hello everyone! Prefect looks awesome — but I’ve been having some issues with flow runs. I am running the agent as a kubernetes pod with

prefect agent start --pool "$PREFECT_ENV" --work-queue default

. 1. It takes really long for the flow to be submitted. Notice the time here is more than 30 seconds for the task to be created, before even running:

15:25:43.676 | INFO    | prefect.agent - Submitting flow run '0dda37d3-87e4-46e2-9266-920e7dae9113'
15:25:44.499 | INFO    | prefect.infrastructure.process - Opening process 'congenial-falcon'...
15:25:44.998 | INFO    | prefect.agent - Completed submission of flow run '0dda37d3-87e4-46e2-9266-920e7dae9113'
<frozen runpy>:128: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
15:26:01.483 | INFO    | Flow run 'congenial-falcon' - Downloading flow code from storage at '/app'
15:26:16.402 | INFO    | Flow run 'congenial-falcon' - Created task run 'return_value-0' for task 'return_value'

2. It downloads the code for every flow, making the pod go out of memory very quickly:

Pod ephemeral local storage usage exceeds the total limit

I am probably missing something here, but would love some help 🙏

QQ for those with experience using GCP's Cloud Run parallelism abilities. I am going to try to run a bunch of commands in parallel, but I am not sure if I should first try to keep everything in Prefect-land and try to use the ``ConcurrentTaskRunner`` and maybe chunk my commands into groups of like 10 (10 would be more than enough).........or if I should try to use the native parallel options available to Cloud Run.

You can structure a job as a single task or as multiple, independent tasks (up to 10,000 tasks) that can be executed in parallel. Each task runs one container instance and can be configured to retry in case of failure. Each task is aware of its index, which is stored in the

CLOUD_RUN_TASK_INDEX

environment variable. The overall count of tasks is stored in the

CLOUD_RUN_TASK_COUNT

environment variable. If you are processing data in parallel, your code is responsible for determining which task handles which subset of the data.

Hi everyone! I’ve started using automations in prefect cloud, and set up an automation that should notify me (pr. email) when a stuck in state “Running”, “Paused”, “Late”, “Resuming” or “Pending” for more than 10 minutes. However, for some reason I occasionally get notifications from this automation when some flow runs completes. Anyone experiencing the same behavior? Any suggestion on how to stop this? Notification example: Flow run get_data entered state

Completed

at 2023-02-22T11:07:44.757581+00:00. Flow ID: x Flow run ID: y Flow run URL: z State message: All states completed. Using prefect 2.8.1

Is there a way to see the flow runs for a flow that occurred more than 7 days ago? We have some flows that only run once a month and can't tell whether they successfully ran last time if it was more than 7 days ago.

Hello everyone Quick question, I want to ask please help me out, I am unable to find out. I have a project named ABC in which flow name is let say XYZ now whenever I run my XYZ flow as everyone knows prefect give random names to the runs. Now I want to get the time at which the latest success run occurred using python. Please help me out. Thanks in advance!

We have a rather urgent problem with Prefect Cloud v1.0. We have zombie tasks that cannot be viewed in the UI, but are queryable via the API and are actively blocking resources. Though we can SEE these zombie tasks we cannot kill them; when we try, we see:

[{'path': ['set_task_run_states'], 'message': "'NoneType' object has no attribute 'flow_id'", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]

Please help.

Take note of the upcoming Prefect Cloud 2 maintenance window:

My agents are not able to push logs to the Prefect API. I’m on 2.7.9. Am I hitting a rate limit for the API?

--- Orion logging error ---
Traceback (most recent call last):
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/logging/handlers.py", line 151, in send_logs
    await client.create_logs(self._pending_logs)
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/client/orion.py", line 1843, in create_logs
    await <http://self._client.post|self._client.post>(f"/logs/", json=serialized_logs)
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_client.py", line 1855, in post
    extensions=extensions,
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/prefect/client/base.py", line 253, in send
    response.raise_for_status()
  File "/root/micromamba/envs/prefect/lib/python3.7/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/><redacted>/workspaces/<redacted>/logs/'

Hi everyone. Is there a problem with site cloud.prefect.io ? Our screen is like that:

image.png

👋 Hi everyone, Heads up - for that past week, we've been chasing down the root cause of an issue causing Cloud 2 automations with a “Stays in” trigger to intermittently misfire. This issue only causes false positives - unwanted triggers - not false negatives - omitted triggers. So while this issue is creating unnecessary noise, the triggers do not miss any cases in which their condition is satisfied. We have made some progress towards a resolution, but we don't have a clear path to a complete fix. We'll provide an update as soon as we're confident that we've resolved the issue. In the meantime, we appreciate your understanding and patience.