https://prefect.io logo
Powered by
Title
t

Tim Helfensdörfer

09/05/2022, 3:40 PM
Hi, we have two work queues, Staging and Production. Exactly the same Deployment files, except the 
work_queue_name
and 
tags
. Both work queues are running at the moment, e.g. with 
prefect agent start  --work-queue "Production"
. But only the Staging work queue picks up jobs. The production work queue does not. It is correctly assigned in the cloud ui, i.e. I can see the late jobs in the work queue tab, but the queue does not get any data: (moved to thread)
✅ 1
a

Anna Geller

09/05/2022, 6:16 PM
tags are deprecated as a mechanism to delegate work to work queues; it works on older versions for backward compatibility, but now, to send your runs from a given deployment to a work queue, you would need to attach the queue name to your deployment e.g. 
prefect deployment build ... -q Staging
for more info, check this comprehensive guide https://discourse.prefect.io/t/prefect-deployments-faq/1467
can you move the code block to the thread? this helps us to keep the main channel cleaner
t

Tim Helfensdörfer

09/06/2022, 7:50 AM
WARNING: Active profile 'production' set by environment variable not found. The default profile will be used instead. 
WARNING: Active profile 'production' set by environment variable not found. The default profile will be used instead. 
Starting agent with ephemeral API...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): Production...
14:59:17.616 | INFO    | prefect.agent - Created work queue 'Production'.
Thanks @Anna Geller, but like I said, we also provided a work queue name. This is the anonymized deployment file: 
###
### A complete description of a Prefect Deployment for flow 'Calculations: Degree Type Order'
###
name: ****-production
description: null
version: ****
# The work queue that will handle this deployment's runs
work_queue_name: Production
tags:
- production
parameters: {}
schedule: null
infra_overrides: {}
infrastructure:
  type: process
  env: {}
  labels: {}
  name: null
  command:
  - python3
  - -m
  - prefect.engine
  stream_output: true

###
### DO NOT EDIT BELOW THIS LINE
###
flow_name: '****'
manifest_path: null
storage: null
path: /app
entrypoint: ****
parameter_openapi_schema:
  title: Parameters
  type: object
  properties: {}
  required: null
  definitions: null
We now updated from 2.1.1 to 2.3.1 and removed the tags. One work queue works, the other one doesn't. It would be great to see some kind of feedback in the web ui. Because clearly the Production work queue is started and the Flow does show in the web interface for this work queue.
a

Anna Geller

09/06/2022, 8:28 AM
Great you got it working. Not sure if this is the issue but you can only supply one queue name to any given deployment. UI page for work queues and linking with deployments is on the roadmap. Thanks for moving the code
t

Tim Helfensdörfer

09/06/2022, 8:30 AM
Sry, there has been a misunderstanding, but it is not working 😕
a

Anna Geller

09/06/2022, 8:34 AM
Can you try without capitalization? Just production
Are you using prefect deployment CLI?
Again I recommend checking the resources here, it should clarify confusion around work queues and deployments relationship and provides many examples on how to make it work https://discourse.prefect.io/t/prefect-deployments-faq/1467
t

Tim Helfensdörfer

09/06/2022, 11:37 AM
The other queue is named "Staging" with a capital S and does work too. We use the same process to create both deployment files, for the Staging and the Production Queue. The only difference is that that one deployment file is named xyz-staging.yaml and the other one xyz-production.yaml. When I compare both files the only difference is the name and the work_queue_name key. But I will check out the ressources again. Just saying that it would really make things easier if some kind of debugging or feedback tool were available. Not just an icon "late".
This is why the manifest file exists - to detect changes that could break a deployment.
Ok the problem was that we forgot to set one environment variable 
PREFECT_API_URL
in our production docker container. Thats why we got the message from above 
Starting agent with ephemeral API...
instead of the message we got in our staging container 
Starting v2.3.1 agent connected to <https://api.prefect.cloud/api/accounts/****/work>
. Thanks anyways.
:thank-you: 1
🙌 1
a

Anna Geller

09/06/2022, 2:12 PM
thanks for the update, great to hear you were able to resolve it
t

Tim Helfensdörfer

09/06/2022, 2:12 PM
But please don't consider this resolved. I don't know how I could have had resolved this issue if we wouldn't already have a working system already.
a

Anna Geller

09/06/2022, 2:15 PM
not sure what you mean. generally, if you need more help with the setup, we have infrastructure experts available in paid support channel via cs@prefect.io - in the community, we can help guide you and point you to the right resources though
t

Tim Helfensdörfer

09/06/2022, 2:24 PM
Ok can you please send me the docs to the ephemeral API or to some explanation on what that is? Can't seem to find it.
a

Anna Geller

09/06/2022, 2:26 PM
it's a local Prefect Orion instance using SQLite DB
it's the default option when you do: 
pip install prefect
prefect orion start
t

Tim Helfensdörfer

09/06/2022, 2:36 PM
Ok maybe I am thinking too complicated but when I read log messages I like to know what's going on. When I read the message above I did not see that the agent was not connected to the cloud api. Yes it said it was connected to the ephemeral API but without searching the docs for every word in the log message I could not figure out directly that something is not working as expected. Ideas: • Another log message: 
Starting v2.3.1 agent connected to local sqlite database /path/to/sqlite
• Flag to confirm that you want to connect to the cloud or the local sqlite database 
prefect agent start --work-queue 'Queue' --cloud
and throw an error if it could not connect to the cloud In the end, for us, it does not matter anymore what you do about this small issue. But we keep finding these small annoyances and just try to make prefect a better product because, and I want to be honest here, if there would be another product with the same functionality we would have switched by now.
And of course, you don't need to care, we aren't paying customers yet. But we just want to help.
a

Anna Geller

09/06/2022, 2:47 PM
We do care! Are you trying to set up CI CD? What is your use case here for setting up this URL?
For local development you can use prefect cloud login CLI
For CI CD you would instead export or set the PREFECT_API_URL variable
I'm a bit confused right now about whether you still have an unresolved issue or whether you are trying to propose a feature request. If the latter is true, it would be easier to create a GitHub issue with a feature request and suggest what feature you would like to propose and why and this makes it easier to communicate with the product team and other engineers
t

Tim Helfensdörfer

09/06/2022, 3:10 PM
Our setup is: Create and apply the deployments for ~30 Flows locally via a custom script, build and deploy docker images via gitlab/aws ecs and then run the queues in ecs fargate. But yeah this is definitely a feature request, i will bring this to GitHub, thanks!
:thank-you: 1
🙌 1
3 Views
#prefect-communityJoin Slack
Powered by