hey everyone, is there a way to use variables that are global to every task like an increment counter ?
✅ 1
Florian Guily
06/09/2022, 1:09 PM
in prefect 1
a
Anna Geller
06/09/2022, 1:25 PM
Good question! you should consider task runs as stateless. In fact, if you leverage DaskExecutor, then each of your task runs can run on a different machine, that's why sharing state between tasks is hard and impractical
you can share some global static variable or pass some value to downstream tasks as data dependency e.g. from a parameter value, but the increment counter-example won't work reliably across executors and various environments
Anna Geller
06/09/2022, 1:26 PM
main point: you shouldn't rely on any shared state between task runs and leverage data dependencies to pass data between tasks
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.