Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
x
Xavier Babu
04/25/2022, 3:44 PMFor some reason, from today onwards in our Orion server, every new execute is going as scheduled by default, not running the flow immediately. Please shed some light.
:discourse: 1
I am running it via REST APIs. If I run it using CLI, it works fine. Is it possible my scheduled.type is messed up?
k
Kevin Kho
04/25/2022, 4:13 PMWhat happens when you preview the work queue?
x
Xavier Babu
04/25/2022, 4:16 PMIt shows "scheduled" in the UI and never runs right away.
k
Kevin Kho
04/25/2022, 4:17 PMDo you have an agent running?
x
Xavier Babu
04/25/2022, 4:17 PMyes
k
Kevin Kho
04/25/2022, 4:20 PMWhat are the tags/settings on the work queue the agent is using?
:upvote: 1
a
Anna Geller
04/25/2022, 4:22 PMScheduled means the agent doesn't pick it up from the work queue - it's likely you configured your work queue for specific tags and you forgot this tag on your DeploymentSpec - could it be?
if not, can you share your
DeploymentSpecx
Xavier Babu
04/25/2022, 4:26 PMok, let me look.
Agent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'
k
Kevin Kho
04/25/2022, 4:59 PMIs it working now?
x
Xavier Babu
04/25/2022, 5:09 PMno
still scheduling all jobs
a
Anna Geller
04/25/2022, 5:11 PMcould you answer my questions?
x
Xavier Babu
04/25/2022, 5:15 PMDeploymentSpec(
flow=custom_flow2,
name="XBparent_flow2_deploy",
tags={'waves_q'},
parameters={'sql': 'SELECT count(*) from table_name;'}
)
If I run in CLI, it runs immediately
k
Kevin Kho
04/25/2022, 5:27 PMWe can’t help you without knowing the work queue configuration
x
Xavier Babu
04/25/2022, 5:28 PMSorry, please let me know how to get the WorkQueue configuration.
k
Kevin Kho
04/25/2022, 5:29 PMIn the UI, click on work queues on the left side and then open the one your agent is using
x
Xavier Babu
04/25/2022, 5:32 PMWhen I click the 3rd button from the top on the left hand side of the UI, it is not showing only the title WorkQueue and nothing more.
a
Anna Geller
04/25/2022, 5:32 PMtags should be a list rather than a dictionary - the brackets are just wrong
tags=["waves_q"]x
Xavier Babu
04/25/2022, 5:34 PMlet me change it and get back to you. Thanks, Anna and Kevin.
👍 1
is_schedule_active=True? What would it do?
Still not working. Even I do QuickRUn from the UI and it shows the state as "late". I haven't added any schedule in this deployment.
a
Anna Geller
04/25/2022, 6:54 PMcan you share more info? perhaps you could describe all steps you took step by step so that we can find out what went wrong
x
Xavier Babu
04/25/2022, 7:56 PMIs it possible to have a zoom call and share it with you? I can deploy the flow with no error and while execute/run via REST APIs, it is going in "scheduled" or "late" state instead of running. But If I run it via CLI like "prefect deployment execute", it runs ok
For some reason, when we deploy from RESTAPI, it is not moving from Submitted to Running State. Instead it goes to Late state.
In another case when I deploy using scheduled in the DeploymentSpec I got the following exception in the agent and agent crashed.
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
Agent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'...
Traceback (most recent call last):
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 747, in get_runs_in_work_queue
response = await self._client.post(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 137, in post
return await self.request(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 80, in request
response.raise_for_status()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_models.py", line 1510, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://dtord03aop02d.dc.dotomi.net:5200/api/work_queues/9a1bc942-6c14-4054-9cb9-c77c1e99fe2b/get_runs'
For more information check: https://httpstatuses.com/404
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/base.py", line 59, in wrapper
return fn(*args, **kwargs)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
return future.result()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/agent.py", line 71, in start
await agent.get_and_submit_flow_runs()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_and_submit_flow_runs
submittable_runs = await self.client.get_runs_in_work_queue(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 753, in get_runs_in_work_queue
raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound
An exception occurred.
a
Anna Geller
04/25/2022, 8:08 PMIs it possible to have a zoom call and share it with you?we don't do that as part of community support, you would need paid support tier - you could reach out to cs@prefect.io can you share the flow code? this spec is fine:
DeploymentSpec(
flow=custom_flow2,
name="XBparent_flow2_deploy",
tags=["waves_q"],
parameters={'sql': 'SELECT count(*) from table_name;'}
)prefect work-queue create somename -t waves_qx
Xavier Babu
04/26/2022, 3:12 PMafter we scheduled a flow, while running agent crashes with the following message.
raise ValueError(f"Unregistered flow runner {typename!r}")
ValueError: Unregistered flow runner 'string'
An exception occurred.
Do we need to register the FlowRunner? What we have to fill-in in flowrunner config()?
I fixed the FlowRunner using Subprocess and working fine now. Thanks Anna and Kevin.
a
Anna Geller
04/26/2022, 5:20 PMThanks for sharing - so it was some sort of typo or misconfiguration in the flow runner? Or were you using some custom subclass of a FlowRunner?
x
Xavier Babu
04/26/2022, 5:21 PMMisconfiguration
But it shouldn't crash the agent every time. Instead it should have thrown some error message to guide us.
a
Anna Geller
04/26/2022, 5:29 PMthanks for sharing and glad it's resolved now