https://prefect.io logo
Title
a

Aman Tripathi

09/04/2022, 5:15 PM
Hi Team, Newbie to 2.0. We are in process to upgrade to 2.0. We are using Prefect + PySpark in our project. In Prefect 1.0 we used resource_manager (the decorator) for SparkContext, so SparkSession is initialized at the beginning of the flow and remains throughout the lifetime of the flow. What is the equivalent of resource_manager decorator in 2.0? Thanks for your help.
1
a

Anna Geller

09/04/2022, 8:42 PM
Hi Aman! Check out this page for more details https://discourse.prefect.io/t/how-to-clean-up-resources-used-in-a-flow/84 but TLDR is you can use any Python context manager directly in your flow to share resources such as Spark context between tasks
a

Aman Tripathi

09/05/2022, 6:48 AM
Got it. Thanks a ton, @Anna Geller
🙌 1