https://prefect.io logo
#prefect-community
Title
# prefect-community
x

Xavier Babu

04/25/2022, 3:44 PM
For some reason, from today onwards in our Orion server, every new execute is going as scheduled by default, not running the flow immediately. Please shed some light.
discourse 1
I am running it via REST APIs. If I run it using CLI, it works fine. Is it possible my scheduled.type is messed up?
k

Kevin Kho

04/25/2022, 4:13 PM
What happens when you preview the work queue?
x

Xavier Babu

04/25/2022, 4:16 PM
It shows "scheduled" in the UI and never runs right away.
k

Kevin Kho

04/25/2022, 4:17 PM
Do you have an agent running?
x

Xavier Babu

04/25/2022, 4:17 PM
yes
k

Kevin Kho

04/25/2022, 4:20 PM
What are the tags/settings on the work queue the agent is using?
upvote 1
a

Anna Geller

04/25/2022, 4:22 PM
Scheduled means the agent doesn't pick it up from the work queue - it's likely you configured your work queue for specific tags and you forgot this tag on your DeploymentSpec - could it be? if not, can you share your
DeploymentSpec
and your work queue and agent configuration?
x

Xavier Babu

04/25/2022, 4:26 PM
ok, let me look.
Agent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'
k

Kevin Kho

04/25/2022, 4:59 PM
Is it working now?
x

Xavier Babu

04/25/2022, 5:09 PM
no
still scheduling all jobs
a

Anna Geller

04/25/2022, 5:11 PM
could you answer my questions?
x

Xavier Babu

04/25/2022, 5:15 PM
DeploymentSpec( flow=custom_flow2, name="XBparent_flow2_deploy", tags={'waves_q'}, parameters={'sql': 'SELECT count(*) from table_name;'} )
If I run in CLI, it runs immediately
k

Kevin Kho

04/25/2022, 5:27 PM
We can’t help you without knowing the work queue configuration
x

Xavier Babu

04/25/2022, 5:28 PM
Sorry, please let me know how to get the WorkQueue configuration.
k

Kevin Kho

04/25/2022, 5:29 PM
In the UI, click on work queues on the left side and then open the one your agent is using
x

Xavier Babu

04/25/2022, 5:32 PM
When I click the 3rd button from the top on the left hand side of the UI, it is not showing only the title WorkQueue and nothing more.
a

Anna Geller

04/25/2022, 5:32 PM
tags should be a list rather than a dictionary - the brackets are just wrong
Copy code
tags=["waves_q"]
x

Xavier Babu

04/25/2022, 5:34 PM
let me change it and get back to you. Thanks, Anna and Kevin.
👍 1
is_schedule_active=True? What would it do?
Still not working. Even I do QuickRUn from the UI and it shows the state as "late". I haven't added any schedule in this deployment.
a

Anna Geller

04/25/2022, 6:54 PM
can you share more info? perhaps you could describe all steps you took step by step so that we can find out what went wrong
x

Xavier Babu

04/25/2022, 7:56 PM
Is it possible to have a zoom call and share it with you? I can deploy the flow with no error and while execute/run via REST APIs, it is going in "scheduled" or "late" state instead of running. But If I run it via CLI like "prefect deployment execute", it runs ok
For some reason, when we deploy from RESTAPI, it is not moving from Submitted to Running State. Instead it goes to Late state.
In another case when I deploy using scheduled in the DeploymentSpec I got the following exception in the agent and agent crashed.
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/contextlib.py", line 153, in exit self.gen.throw(typ, value, traceback) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ConnectError: All connection attempts failed An exception occurred.
Agent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'... Traceback (most recent call last): File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 747, in get_runs_in_work_queue response = await self._client.post( File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 137, in post return await self.request( File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 80, in request response.raise_for_status() File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_models.py", line 1510, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://dtord03aop02d.dc.dotomi.net:5200/api/work_queues/9a1bc942-6c14-4054-9cb9-c77c1e99fe2b/get_runs' For more information check: https://httpstatuses.com/404 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/base.py", line 59, in wrapper return fn(*args, **kwargs) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper return run_async_in_new_loop(async_fn, *args, **kwargs) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop return anyio.run(partial(__fn, *args, **kwargs)) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run return asynclib.run(func, *args, **backend_options) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run return native_run(wrapper(), debug=debug) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete return future.result() File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper return await func(*args) File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/agent.py", line 71, in start await agent.get_and_submit_flow_runs() File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_and_submit_flow_runs submittable_runs = await self.client.get_runs_in_work_queue( File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 753, in get_runs_in_work_queue raise prefect.exceptions.ObjectNotFound(http_exc=e) from e prefect.exceptions.ObjectNotFound An exception occurred.
a

Anna Geller

04/25/2022, 8:08 PM
Is it possible to have a zoom call and share it with you?
we don't do that as part of community support, you would need paid support tier - you could reach out to cs@prefect.io can you share the flow code? this spec is fine:
Copy code
DeploymentSpec(
    flow=custom_flow2,
    name="XBparent_flow2_deploy",
    tags=["waves_q"],
    parameters={'sql': 'SELECT count(*) from table_name;'}
)
provided that you have a work queue with this tag:
Copy code
prefect work-queue create somename -t waves_q
x

Xavier Babu

04/26/2022, 3:12 PM
after we scheduled a flow, while running agent crashes with the following message. raise ValueError(f"Unregistered flow runner {typename!r}") ValueError: Unregistered flow runner 'string' An exception occurred. Do we need to register the FlowRunner? What we have to fill-in in flowrunner config()?
I fixed the FlowRunner using Subprocess and working fine now. Thanks Anna and Kevin.
a

Anna Geller

04/26/2022, 5:20 PM
Thanks for sharing - so it was some sort of typo or misconfiguration in the flow runner? Or were you using some custom subclass of a FlowRunner?
x

Xavier Babu

04/26/2022, 5:21 PM
Misconfiguration
But it shouldn't crash the agent every time. Instead it should have thrown some error message to guide us.
a

Anna Geller

04/26/2022, 5:29 PM
thanks for sharing and glad it's resolved now
8 Views