Hi, I have an issue: I have a task, where basically constitutes of polling an API every 30 seconds. ...

Tadej Svetina

01/08/2023, 9:28 PM

Hi, I have an issue: I have a task, where basically constitutes of polling an API every 30 seconds. I use

asyncio.sleep

to wait between querying the API. Running of this task occupies 100% of the CPU - how can I fix this?

👀 1

Tadej Svetina

01/08/2023, 9:51 PM

Ok, I have a nice replication example. This occurs only when using

submit

- so there must be a problem with the task runner.

Tadej Svetina

01/08/2023, 9:51 PM

Copy code

import asyncio
import prefect


@prefect.task
async def sleepy():
    while True:
        print("Hi")
        await asyncio.sleep(10)


@prefect.flow
async def sleepy_flow():
    await sleepy.submit()

asyncio.run(sleepy_flow())

Tadej Svetina

01/08/2023, 9:51 PM

Without using submit, I am getting normal (near 0) CPU usage

George Coyne

01/10/2023, 3:50 PM

https://github.com/python/asyncio/issues/398

George Coyne

01/10/2023, 3:50 PM

It looks like this is expected behavior

Tadej Svetina

01/10/2023, 4:13 PM

This is not the same - the issue you posted uses asyncio.sleep(0), so there would be constant polling. I use asyncio.sleep(10), which means that the event loop should simply ignore the task for 10 seconds

Zanie

01/10/2023, 4:13 PM

👍 yeah this is not proper behavior — I’m changing how we handle concurrency so hopefully it is addressed by that.

Zanie

01/10/2023, 4:15 PM

It seems possible that we’re encountering one of the spin locks at https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_runners.py#L317-L327

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.