For some reason, from today onwards in our Orion s...
# ask-communityx
Xavier Babu
04/25/2022, 3:44 PMFor some reason, from today onwards in our Orion server, every new execute is going as scheduled by default, not running the flow immediately. Please shed some light.
discourse 1
Xavier Babu
04/25/2022, 3:54 PMI am running it via REST APIs. If I run it using CLI, it works fine. Is it possible my scheduled.type is messed up?
k
Kevin Kho
04/25/2022, 4:13 PM
What happens when you preview the work queue?
x
Xavier Babu
04/25/2022, 4:16 PMIt shows "scheduled" in the UI and never runs right away.
k
Kevin Kho
04/25/2022, 4:17 PM
Do you have an agent running?
x
Xavier Babu
04/25/2022, 4:17 PMyes
k
Kevin Kho
04/25/2022, 4:20 PM
What are the tags/settings on the work queue the agent is using?
upvote 1
a
Anna Geller
04/25/2022, 4:22 PM
Scheduled means the agent doesn't pick it up from the work queue - it's likely you configured your work queue for specific tags and you forgot this tag on your DeploymentSpec - could it be?
if not, can you share your
DeploymentSpecx
Xavier Babu
04/25/2022, 4:26 PMok, let me look.
Xavier Babu
04/25/2022, 4:28 PMAgent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'
k
Kevin Kho
04/25/2022, 4:59 PM
Is it working now?
x
Xavier Babu
04/25/2022, 5:09 PMno
Xavier Babu
04/25/2022, 5:09 PMstill scheduling all jobs
a
Anna Geller
04/25/2022, 5:11 PM
could you answer my questions?
x
Xavier Babu
04/25/2022, 5:15 PMDeploymentSpec(
flow=custom_flow2,
name="XBparent_flow2_deploy",
tags={'waves_q'},
parameters={'sql': 'SELECT count(*) from table_name;'}
)
Xavier Babu
04/25/2022, 5:25 PMIf I run in CLI, it runs immediately
k
Kevin Kho
04/25/2022, 5:27 PM
We can’t help you without knowing the work queue configuration
x
Xavier Babu
04/25/2022, 5:28 PMSorry, please let me know how to get the WorkQueue configuration.
k
Kevin Kho
04/25/2022, 5:29 PM
In the UI, click on work queues on the left side and then open the one your agent is using
x
Xavier Babu
04/25/2022, 5:32 PMWhen I click the 3rd button from the top on the left hand side of the UI, it is not showing only the title WorkQueue and nothing more.
a
Anna Geller
04/25/2022, 5:32 PM
tags should be a list rather than a dictionary - the brackets are just wrong
Copy code
tags=["waves_q"]x
Xavier Babu
04/25/2022, 5:34 PMlet me change it and get back to you. Thanks, Anna and Kevin.
👍 1
Xavier Babu
04/25/2022, 6:14 PMis_schedule_active=True? What would it do?
Xavier Babu
04/25/2022, 6:47 PMStill not working. Even I do QuickRUn from the UI and it shows the state as "late". I haven't added any schedule in this deployment.
a
Anna Geller
04/25/2022, 6:54 PM
can you share more info? perhaps you could describe all steps you took step by step so that we can find out what went wrong
x
Xavier Babu
04/25/2022, 7:56 PMIs it possible to have a zoom call and share it with you? I can deploy the flow with no error and while execute/run via REST APIs, it is going in "scheduled" or "late" state instead of running. But If I run it via CLI like "prefect deployment execute", it runs ok
Xavier Babu
04/25/2022, 7:56 PMFor some reason, when we deploy from RESTAPI, it is not moving from Submitted to Running State. Instead it goes to Late state.
Xavier Babu
04/25/2022, 7:57 PMIn another case when I deploy using scheduled in the DeploymentSpec I got the following exception in the agent and agent crashed.
Xavier Babu
04/25/2022, 7:57 PMFile "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
Xavier Babu
04/25/2022, 8:00 PMAgent started! Looking for work from queue '9a1bc942-6c14-4054-9cb9-c77c1e99fe2b'...
Traceback (most recent call last):
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 747, in get_runs_in_work_queue
response = await self._client.post(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 137, in post
return await self.request(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/httpx.py", line 80, in request
response.raise_for_status()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/httpx/_models.py", line 1510, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://dtord03aop02d.dc.dotomi.net:5200/api/work_queues/9a1bc942-6c14-4054-9cb9-c77c1e99fe2b/get_runs'
For more information check: https://httpstatuses.com/404
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/base.py", line 59, in wrapper
return fn(*args, **kwargs)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 120, in wrapper
return run_async_in_new_loop(async_fn, *args, **kwargs)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncio.py", line 67, in run_async_in_new_loop
return anyio.run(partial(__fn, *args, **kwargs))
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
return future.result()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/cli/agent.py", line 71, in start
await agent.get_and_submit_flow_runs()
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_and_submit_flow_runs
submittable_runs = await self.client.get_runs_in_work_queue(
File "/localpart0/aop-shared/WAVES/prefect2/lib/python3.10/site-packages/prefect/client.py", line 753, in get_runs_in_work_queue
raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound
An exception occurred.
a
Anna Geller
04/25/2022, 8:08 PM
Is it possible to have a zoom call and share it with you?we don't do that as part of community support, you would need paid support tier - you could reach out to cs@prefect.io can you share the flow code? this spec is fine:
Copy code
DeploymentSpec(
flow=custom_flow2,
name="XBparent_flow2_deploy",
tags=["waves_q"],
parameters={'sql': 'SELECT count(*) from table_name;'}
) Copy code
prefect work-queue create somename -t waves_qx
Xavier Babu
04/26/2022, 3:12 PMafter we scheduled a flow, while running agent crashes with the following message.
raise ValueError(f"Unregistered flow runner {typename!r}")
ValueError: Unregistered flow runner 'string'
An exception occurred.
Do we need to register the FlowRunner? What we have to fill-in in flowrunner config()?
Xavier Babu
04/26/2022, 4:36 PMI fixed the FlowRunner using Subprocess and working fine now. Thanks Anna and Kevin.
a
Anna Geller
04/26/2022, 5:20 PM
Thanks for sharing - so it was some sort of typo or misconfiguration in the flow runner? Or were you using some custom subclass of a FlowRunner?
x
Xavier Babu
04/26/2022, 5:21 PMMisconfiguration
Xavier Babu
04/26/2022, 5:21 PMBut it shouldn't crash the agent every time. Instead it should have thrown some error message to guide us.
a
Anna Geller
04/26/2022, 5:29 PM
thanks for sharing and glad it's resolved now
Anna Geller
04/26/2022, 5:40 PM
8 Views