Hi all, my team and I are developing a web-app to load and process dataframes, we're willing to use prefect orion instead of jupyter kernls. Is it possible to separate contexts/sessions using prefect? Example: User A creates a variable "var_a" and User B creates a variable "var_b". User A shouldn't be able to access to "var_b". Both variables should be accessible trough several flows from the same user until their session on our app is closed. Am I missing something?

I think each Flow will have it’s own separate process so I don’t know how you would share that variable unless you explicitly load it in somehow

Thanks, is there any recommendation to achieve that?

Save it in a file in an S3 bucket or something equivalent and then load it in like you would any other piece of data. Or! I realized maybe you can pass the variable as a parameter when you kick off each Flow Run. How big is the data?

We manage large datasets (it depends on the user and on the infrastructure) but we expect to make several requests and modifications in short lapses of time, using S3 could slow us down.

It sounds like you have a “temporary database” that you are making a lot of requests towards and want to make it available to multiple Flows? I don’t see how you can make it shared in-memory without persisting because each Flow is a separate process. Results are passed from sub-flows to main flows through persistence because they are on separate executors. Are you transformations OLAP and SQL-like? Cuz I would look into DuckDB. If it’s more transaction-heavy, this is very hard