Hi everyone, I've run into what looks like a bug in Prefect Cloud. No tasks are running, yet this State Message is being reported:

Queued due to concurrency limits. The local process will attempt to run the task for the next 10 minutes, after which time it will be made available to other agents.

That string does not appear in the open-source part of Prefect, so it must be part of Prefect Cloud. The concurrency limit on that task is 10, and things were working until I changed some of Dask's configuration parameters to try to resolve an issue with it. The most likely cause of the above message is that some error happened and it didn't get handled correctly. https://cloud.prefect.io/stockwell/flow-run/4460703d-3c91-4573-b85c-a4b001048999

Hey @Jeremy Phelps, could you try querying for the relevant tags and seeing if there are task runs in a running state? Something like:

query{
  task (where: {tags: {_eq: []}}) {
    flow {
        id
        name
      }
    id
    name
    tags
    task_runs (where: {state: {_eq:"Running"}}) {
      id
      name
      state
    }
    }
  }

I filled in that query as:

query{
  task (where: {tags: {_eq: ["staging"]}}) {
    flow {
        id
        name
      }
    id
    name
    tags
    task_runs (where: {state: {_eq:"Running"}}) {
      id
      name
      state
    }
    }
  }

...and it returned no tasks.

Hi @Jeremy Phelps - can you run this query instead?

query{
  task (where: {tags: {_eq: ["staging"]}}) {
    id
    name
    tags
    task_runs (where: {state: {_in:["Running", "Submitted", "Queued", "Cancelling", "Retrying", "Resume", "Paused"]}}) {
      id
      name
      state
    }
    }
  }

That also returns nothing.

Interesting... let me dig around and see what I can find

Taking off all the parameters after the

task

token returns something, but Slack won't let me send it.

And

staging

is the tag you're having issues with, yeah? (Prefect employees can't see your UI links, just fyi)

Yes.

@Jeremy Phelps could you clarify what you mean?

When I run the GQL query you suggest, that performs a lookup in a database that Prefect owns. Can Prefect employees see what's in that database?

Prefect does have access, you're correct. Here's what I found: The only concurrency limit that's set is 

10

 on a tag called 

mysql-write

, and you already have 10 tasks in a 

running

 state with that tag; the tasks in the flow run you provided also have the 

mysql-write

 tag and are queued correctly as a result. There are no tasks with a

staging

tag and no concurrency limits with that tag either. One thing I did notice is that the tasks in

running

states don't all come from the same flow run, which could be causing the confusion

These tasks are not actually running. How do I find and get rid of them?

Let me see if I can grab some flow run ids and names for you and you can manually mark them as finished or cancelled

That doesn't solve the problem going forward. Things will reach this state again.

Two things you can do for the future: you can set up flow SLA automations for that flow that will fail the flow if it exceeds some time threshold, you can manually mark those flow runs that are holding onto concurrency slots but whose jobs are stale as failed/completed (along with their associated tasks). Basically you'll need to kill the jobs in some way to make sure they're not holding onto those slots, whether that's through Prefect or your cluster

It seems that the only way to have a truly separate staging environment is to have a separate Prefect account for it.

I think what you're describing is entirely doable with a single tenant but separate tags and labels on your flows to account for execution submission to different environments but I can put you in touch with one of our account managers to discuss multi-tenancy which will give you database-level sharding of environments.

Does multi-tenancy cost additional money?

It does, it's an enterprise-grade feature

Management will never agree to it.

The GraphQL api has docs attached to the schema (you can view these in the interactive api) which denotes all input and output types. You can run that mutation through the interactive API like this:

mutation {
  set_task_run_states(input: {states: [{task_run_id: "<<task run states>>", state: "{\"type\": \"Failed\", \"message\": \"<<your message>>\"}"}]}) {
    states {
      id
      status
    }
  }
}

The problem I'm running into is that I don't know which fields are expected in the

state

.