Hi everyone! Has anyone experienced issues with Prefect Automations today? Prefect Cloud status page (status.prefect.io) doesn't show any issues for today We're seeing false alerts in one of our environments.

@Marvin we're running azure push work pools and recently started to get flows moving into state Late instead of retry. Any suggestions how this can be fixed?

A

Late

state usually means the run wasn’t started before its scheduled time, so retry settings won’t apply because the flow hasn’t actually failed yet. Since this is an Azure push work pool, I’d check the work-pool concurrency limits, Azure ACI quotas/capacity, and the Azure Activity Log for authentication or container-creation errors. Did anything recently change with the service principal, managed identity, work-pool template, or Azure subscription limits? If you can share one affected run’s infrastructure events, we can narrow it down.

Hi Community, Is there an official Partner program or someone in partnerships that I can speak to? I’m working with a Japanese systems integrator that is exploring emerging data and orchestration platforms and we would love to help bring Prefect to our customers

Hi @Marvin, On a self hosted prefect OSS v3.6.20, running on k8s, I have a Prefect flow that last 10 min. But the Prefect UI show it last 5 min. After 5 min, the flow is indicated in a crashed state. But no flow/task/subflow seem affected and it finishes in a Completed state. I don't see any high resource consumptions. After 5 min, the flow tries to fetch data from an API but it gets error 429, then sleep during 5 sec (using tenacity) then try again. Before the 5 min, the Prefect logs show :

Timeout of 7200.0 seconds configured for this task, but the task is running in a worker thread. Timeouts in worker threads cannot interrupt blocking operations like `time.sleep()`, network requests, or file I/O. The timeout will only take effect after the blocking operation completes. Consider using an async task with `await` statements for reliable timeout behavior.

Why Prefect show this flow as Crashed while nothing actually crash ?

Hello I am senior software engineer Just looking for job opportunites

We have some issues with Prefect platform now, does anyone has the same? @Marvin

@Kirill Egorov Yes, here is the message from prefect support - We are seeing an increase in 503s in our monitoring and the team is investigating. I will update once I know anything.

@Kirill Egorov Yes we are facing issues with prefect as well, 503s error

Yes, we too.

Issue resolved from prefect side

@Marvin we are using prefect self-hosted on gke. our work pools are healthy, but the work queues are marked as not ready. how can we fix this and what could cause the error?

Anyone know what the cause of the 503 errors were yesterday? And what steps were taken to resolve them?

Anyone facing Prefect Cloud API connectivity issue?

What can we do with prefect cloud? my flows are not running!

Now that service is back i can see the message:

On July 25th at 21:00 UTC, Prefect Cloud will undergo routine scheduled maintenance for our database instances. During this time, the Prefect Cloud APIs will be inaccessible for a brief period, which is typically less than ten minutes, but may last for up to an hour. While work may run late during this maintenance window, no work or data will be lost.

If someone from prefect is listening to this channel, in the future, I would be very happy to see the info above (scheduled maintenance) on status.prefect.io ; and also a link to status.prefect.io from prefect.io site - it took me too long to figure out what was happening.

This is an absolute joke. Scheduled maintenance brings down everything, outages brings down everything and this is happening WAY too often. It's like every second week we have to go in and cancel all our flows manually one by one as they just get stuck in a running or pending state.

@Janet Carson split thread 🧵

Hi! Could I get someone to take a quick look at github.com/conda-forge/prefect-feedstock/pull/454 to make the latest prefect version 3.8.0 available on Conda. Thanks!

Continuing the maintenance conversations -- how are folks handling maintenance windows? Do you shut down your deployments in advance or just log in afterwards and recover?

Hi there! I added a zombie flow detection automation

{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.id": [
      "prefect.flow.5dec07f4-f468-4cdf-99a3-5016aa05c383",
      "prefect.flow.acdba04f-fc39-4e67-990b-b9ff25947e24",
      "prefect.flow.baebc35a-bcaf-47ab-9819-a80f873d88b5",
      "prefect.flow.400db30b-8136-48b0-93bf-f391bded4377",
      "prefect.flow.d0b6d613-82d4-42a4-b2c0-113b186d817b",
      "prefect.flow.ee31395e-3769-4db5-a72c-6fe6a0470e1a",
      "prefect.flow.50bc3efe-2c90-4d06-b82b-08db0faf1a8b",
      "prefect.flow.b992f437-56c4-4644-90a8-4ef6305993c4",
      "prefect.flow.293b63ca-651d-4e8a-b518-9c63d124bf5a"
    ],
    "prefect.resource.role": "flow"
  },
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.*"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 600
}

For some reason, a couple minutes after the flow run completes successfully, I can see the flow still triggers, which I would have not expected. I can see a

prefect.flow-run.Completed

event for one flow at 124950 EST and the automation still triggered at 125807 EST for that particular flow. What am I doing wrong? I feel like this is a Prefect Cloud bug 🤔

Hi everyone

@flow(log_prints=True)
async def load_wso_industry_codes(as_of_date: date | None = None) -> None:
    as_of_date = date.today() - timedelta(days=1) if as_of_date is None else as_of_date
    issuer_ids = get_issuer_ids(as_of_date)
    # print(f'done getting issuer id from db. Count: {len(issuer_ids)}')
    sem = asyncio.Semaphore(20) # avoid blasting WSO server, 20 is sufficiently fast
    async with httpx.AsyncClient(timeout=30) as client:
        token = await get_wso_token(client)
        client.headers['Authorization'] = f"Bearer {token}"

        issuers = await work(sem, client, issuer_ids, wait_for=[issuer_ids])

        print('saving industry codes')
        await save_industry_codes(as_of_date, issuers, wait_for=[issuers])

why my prefect dependency graph is showing a dependency arrow betwen save_industry_codes and get_issuer_ids? also I have to explicitly add

wait_for

for it to show the dependency arrows. It should automatically discover the dependency since I'm passing the result of a task to another one already...

Hi! I’m trying to upgrade to the Prefect Horizon Enterprise plan because I’d like to use a custom domain. I contacted sales@prefect.io, but the automated reply directed me to the contact form, and the form keeps flagging my submission as a test or fake request. Could someone from the Prefect team help me get in touch with sales or the right person for the upgrade? Thanks!

How do you guys monitor for file arrival to trigger a job in Prefect. Do you have long-running or frequently occuring deployments, or use external tools like crontab or systemd services with watchdog to trigger Prefect flows via API?

Hi, I’m having an issue provisioning a custom domain for my Horizon MCP server. The certificate validation keeps failing. Could someone from the Horizon team help me check this? I can share the domain and DNS configuration privately via DM.

Hi, Is there anyone who will work for our startup bussiness?

How are you guys handling the dev/prod workflow with Prefect Cloud and a single workspace? Right now we run most dev workflows locally and send the logs to Prefect Cloud but some of the stuff needs to run in dev but on the cloud (for access/resource reasons). We upload those deployments with proper tags and use a dedicated dev worker for it but I'm not sure if it's optimal or not.

I am going through a HITRUST r2 Certification and one evidence request is to show the Session Timeout Settings for Prefect. Is there a default config settings I can see in my portal? If yes, please point me in the ight direction to view. Thanks.