https://prefect.io logo
Join Slack
Powered by
Hi, we have two work queues, Staging and Productio...
# prefect-community
t
Hi, we have two work queues, Staging and Production. Exactly the same Deployment files, except the 
work_queue_name
and 
tags
. Both work queues are running at the moment, e.g. with 
prefect agent start  --work-queue "Production"
. But only the Staging work queue picks up jobs. The production work queue does not. It is correctly assigned in the cloud ui, i.e. I can see the late jobs in the work queue tab, but the queue does not get any data: (moved to thread)
✅ 1
a
tags are deprecated as a mechanism to delegate work to work queues; it works on older versions for backward compatibility, but now, to send your runs from a given deployment to a work queue, you would need to attach the queue name to your deployment e.g.
Copy code
prefect deployment build ... -q Staging
for more info, check this comprehensive guide https://discourse.prefect.io/t/prefect-deployments-faq/1467
can you move the code block to the thread? this helps us to keep the main channel cleaner
t
Copy code
WARNING: Active profile 'production' set by environment variable not found. The default profile will be used instead. 
WARNING: Active profile 'production' set by environment variable not found. The default profile will be used instead. 
Starting agent with ephemeral API...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): Production...
14:59:17.616 | INFO    | prefect.agent - Created work queue 'Production'.
Thanks @Anna Geller, but like I said, we also provided a work queue name. This is the anonymized deployment file:
Copy code
###
### A complete description of a Prefect Deployment for flow 'Calculations: Degree Type Order'
###
name: ****-production
description: null
version: ****
# The work queue that will handle this deployment's runs
work_queue_name: Production
tags:
- production
parameters: {}
schedule: null
infra_overrides: {}
infrastructure:
  type: process
  env: {}
  labels: {}
  name: null
  command:
  - python3
  - -m
  - prefect.engine
  stream_output: true

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: '****'
manifest_path: null
storage: null
path: /app
entrypoint: ****
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
We now updated from 2.1.1 to 2.3.1 and removed the tags. One work queue works, the other one doesn't. It would be great to see some kind of feedback in the web ui. Because clearly the Production work queue is started and the Flow does show in the web interface for this work queue.
a
Great you got it working. Not sure if this is the issue but you can only supply one queue name to any given deployment. UI page for work queues and linking with deployments is on the roadmap. Thanks for moving the code
t
Sry, there has been a misunderstanding, but it is not working 😕
a
Can you try without capitalization? Just production
Are you using prefect deployment CLI?
Again I recommend checking the resources here, it should clarify confusion around work queues and deployments relationship and provides many examples on how to make it work https://discourse.prefect.io/t/prefect-deployments-faq/1467
t
The other queue is named "Staging" with a capital S and does work too. We use the same process to create both deployment files, for the Staging and the Production Queue. The only difference is that that one deployment file is named xyz-staging.yaml and the other one xyz-production.yaml. When I compare both files the only difference is the name and the work_queue_name key. But I will check out the ressources again. Just saying that it would really make things easier if some kind of debugging or feedback tool were available. Not just an icon "late".
This is why the manifest file exists - to detect changes that could break a deployment.
Ok the problem was that we forgot to set one environment variable 
PREFECT_API_URL
in our production docker container. Thats why we got the message from above 
Starting agent with ephemeral API...
instead of the message we got in our staging container 
Starting v2.3.1 agent connected to <https://api.prefect.cloud/api/accounts/****/work>
. Thanks anyways.
🙏 1
🙌 1
a
thanks for the update, great to hear you were able to resolve it
t
But please don't consider this resolved. I don't know how I could have had resolved this issue if we wouldn't already have a working system already.
a
not sure what you mean. generally, if you need more help with the setup, we have infrastructure experts available in paid support channel via cs@prefect.io - in the community, we can help guide you and point you to the right resources though
t
Ok can you please send me the docs to the ephemeral API or to some explanation on what that is? Can't seem to find it.
a
it's a local Prefect Orion instance using SQLite DB
it's the default option when you do:
Copy code
pip install prefect
prefect orion start
t
Ok maybe I am thinking too complicated but when I read log messages I like to know what's going on. When I read the message above I did not see that the agent was not connected to the cloud api. Yes it said it was connected to the ephemeral API but without searching the docs for every word in the log message I could not figure out directly that something is not working as expected. Ideas: • Another log message: 
Starting v2.3.1 agent connected to local sqlite database /path/to/sqlite
• Flag to confirm that you want to connect to the cloud or the local sqlite database 
prefect agent start --work-queue 'Queue' --cloud
and throw an error if it could not connect to the cloud In the end, for us, it does not matter anymore what you do about this small issue. But we keep finding these small annoyances and just try to make prefect a better product because, and I want to be honest here, if there would be another product with the same functionality we would have switched by now.
And of course, you don't need to care, we aren't paying customers yet. But we just want to help.
a
We do care! Are you trying to set up CI CD? What is your use case here for setting up this URL?
For local development you can use prefect cloud login CLI
For CI CD you would instead export or set the PREFECT_API_URL variable
I'm a bit confused right now about whether you still have an unresolved issue or whether you are trying to propose a feature request. If the latter is true, it would be easier to create a GitHub issue with a feature request and suggest what feature you would like to propose and why and this makes it easier to communicate with the product team and other engineers
t
Our setup is: Create and apply the deployments for ~30 Flows locally via a custom script, build and deploy docker images via gitlab/aws ecs and then run the queues in ecs fargate. But yeah this is definitely a feature request, i will bring this to GitHub, thanks!
🙏 1
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by