Tom Matthews
07/04/2022, 9:16 AMthreads
scheduler? 🙏Ron Levi
07/04/2022, 9:25 AMdef async_task(fn=None, **kwargs):
if fn is None:
return lambda fn: async_task(fn, **kwargs)
@task(**kwargs)
@wraps(fn)
def inner(*args, **kwargs):
asyncio.run(fn(*args, **kwargs))
return inner
Tom Matthews
07/04/2022, 9:35 AMimport asyncio
from functools import wraps
from prefect import Flow, task
def async_task(fn=None, **kwargs):
@task(**kwargs)
@wraps(fn)
def inner(*args, **kwargs):
asyncio.run(fn(*args, **kwargs))
return inner
@async_task
async def count():
print("One")
await asyncio.sleep(1)
print("Two")
@async_task
async def main():
return await asyncio.gather(count(), count(), count())
async def run_flow():
async with Flow("Test") as flow:
result = await main()
flow.run()
if __name__ == "__main__":
asyncio.run(run_flow())
Can you see what i’m doing wrong here? Also, im pretty new to asyncio 😅Ron Levi
07/04/2022, 10:00 AMTom Matthews
07/04/2022, 11:07 AMAnna Geller
07/04/2022, 11:58 AMtasks that are essentially API calls which can be run in parallel and take ~ 1 second each but every second of latency saved is very importantseems like a great use case for ConcurrentTaskRunner
prefect 2.0 and that is not recommended in productionit's not as clear-cut - it depends on your use case and what is meant by production really. Check this page for more details - and GA of Prefect 2.0 is just around the corner so I highly encourage you to use Prefect 2.0 for that use case rather than building workarounds with Prefect 1.0 now
Ron Levi
07/04/2022, 12:02 PMAnna Geller
07/04/2022, 12:03 PMwould it be safe to use prefect 2 in ProductionPrefect 2.0 is an open-source product and with that, you have the same guarantees as with any OSS product - you are in charge of making that production-ready yourself. However, for Cloud 2.0 we do our best effort to avoid any data loss and when we introduce any breaking changes to the API, we always provide instructions on how you can adjust your flows or execution layer setup to match with the latest release, so it's not as scary as it seems to be. But for sure, you need to judge yourself based on your appetite for software and your use case/needs in your team
Tom Matthews
07/04/2022, 2:33 PMConcurrentTaskRunner
and leveraging the native support for async concurrency? Would you make use of asyncio
if you had concerns around thread safety?Anna Geller
07/04/2022, 2:47 PM