Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Is it possible to define async tasks in Python using `async def...` ?

As far as I know, this is not possible in Prefect 1.0 but will be in Prefect 2.0 aka Orion. You should already be able to try out the technical preview if you'd like to but Orion is not yet production ready.

Honestly, I am a bit disappointed with the maturity of version 0.15, which we started with just a week or more before the 1.0 was released. Checking the 1.0 now.

*1) Regarding async in Orion:* 
• as Sylvain pointed out, Orion supports async tasks and flows out of the box. 
• Orion is planned to come out of the technical preview into a beta release by the end of this month so it's a perfect timing for you to try Orion with async workflows
*2) Regarding async in Prefect &lt;= 1.0:*
Slack deleted the thread by I have notes from a person who asked about that.

Question: When working on a task that needs to download images for specific lat/lon, how to combine Mapping with async to process those images efficiently?

Answer from the community user that uses async in Prefect 1.0:
• I think prefect can handle maps over millions of items well, as long as the underlying dask cluster is sufficiently sized.
• you need to utilize `async` strictly *inside your task*s. Split your lat-lon pairs into, say, batches of size 100. Then *every task can download its own batch of size 100 asynchronously*. 
• prefects unit of work is a task. And dask won't run these tasks asynchronously, so async isn't a viable option to concurrently execute a lot of prefect tasks. But *inside the task async can be utilized*, no issues.
• note that task Results are held in memory after flows complete and having a high number of futures in Dask can be indeed too memory-intensive so batching would help with that

Hey <@U02H1A95XDW> is there an example of this?

No, that's all I got from the community user. But I believe doing it directly in Orion is more beneficial, especially long-term

Yeah I was just trying to see what my options were for today as I have to demo this to my stakeholders and currently the file takes 5 minutes to run

Just out of curiosity, which parts of v1 feel the most immature?

I was talking about 0.15.* I found, what I believe is a bug regarding task timeout via a decorator, and a minor Web UI bug, when I click on tab overview/logs/e.t.c. using the 3rd button of the mouse (open in new tab) the URL has a `#` instead of the `?` at the end, so front-end does not render a thing