https://prefect.io logo
#prefect-community
Title
# prefect-community
t

Tadej Svetina

01/08/2023, 9:28 PM
Hi, I have an issue: I have a task, where basically constitutes of polling an API every 30 seconds. I use
asyncio.sleep
to wait between querying the API. Running of this task occupies 100% of the CPU - how can I fix this?
👀 1
Ok, I have a nice replication example. This occurs only when using
submit
- so there must be a problem with the task runner.
Copy code
import asyncio
import prefect


@prefect.task
async def sleepy():
    while True:
        print("Hi")
        await asyncio.sleep(10)


@prefect.flow
async def sleepy_flow():
    await sleepy.submit()

asyncio.run(sleepy_flow())
Without using submit, I am getting normal (near 0) CPU usage
g

George Coyne

01/10/2023, 3:50 PM
It looks like this is expected behavior
t

Tadej Svetina

01/10/2023, 4:13 PM
This is not the same - the issue you posted uses asyncio.sleep(0), so there would be constant polling. I use asyncio.sleep(10), which means that the event loop should simply ignore the task for 10 seconds
z

Zanie

01/10/2023, 4:13 PM
👍 yeah this is not proper behavior — I’m changing how we handle concurrency so hopefully it is addressed by that.
It seems possible that we’re encountering one of the spin locks at https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_runners.py#L317-L327
3 Views