Hi there, im trying to build a flow where at the beginning of the flow I have a task that creates a new database then I want to pass that to all of the remaining tasks. (ie one task preps the stage and then all other tasks use that) Is there a way to do this with the context? Can have that task write to the context and then have all my other tasks reference that?

AFAIK a task cannot write something to the prefect context for the downstream tasks to read from it, it has to be a task output. However, depending on your use case I may have an idea. You can write to the prefect context before the flow run starts. Write something to the context that can be used to uniquely identify the database to be created. Then your root task creates the database by reading from the context, and other tasks derive the database by reading from the context.

Hi @joshua mclellan, You can do this with the new KV Store! Write the location to the KV Store, and then connect it to it in downstream tasks. Docs

@emre thats a fantastic workaround, if they didnt have a new feature that addresses this specifically thats definitely what i would have done

Yeah that’s really clever

@Kevin Kho the docs dont have a python example for the python version of setting key-value pairs. not blocking be since the function is used in the flows example but probably a minor ticket for someone somewhere

So I brought that up too, but to be honest there’s something there and the docs is just not rendering it for some people but it is for others 😅. I know what you mean though.

weird - so it looks like kv-stores are only for the prefect cloud back end

You mean you’re on server and not using Prefect Cloud? There would not be. The KVStore is online only. You’d need to do something like emre’s suggestion or pass the references to that database downstream to tasks

yea just running things locally on a LocalExecutor

LocalExecutor can still be backed by PrefectCloud

do you have docs that I could use to read up a bit on this. im not entirely sure i understand why you would use the local executor on the cloud setting. We are only using it over dask because some of our tasks can be really large and we didnt want to deal with the headache of two large tasks running at once but that isnt backed by a tonn of investigation on my part

No docs that specifically clears up the concepts but let’s break it down. Prefect Cloud and Prefect Server are

backends

. Cloud is the one you use at

<http://cloud.prefect.io|cloud.prefect.io>

, Server is when you spin it up on your own infrastructure. The backend is the API you use so whatever Flow runs uses the backend to hit the API and record the states of execution. Agents also ping the backend API to pick up flows. The LocalExecutor is an

Executor

. Executors are there to determine how the Flow is run by the Agent once it’s retrieved. The LocalExecutor just means a sequential single-threaded executor environment. It is called local because it does not make any external connections to run the Flow. Similarly, the LocalDaskExecutor uses a local Dask multiprocessing pool to execute the Flow. This parallelizes task execution. Again local means they don’t interact with other machines to execute. On the other hand, the DaskExecutor is a

distributed

executor and executes the Flow over a cluster of machines. That’s why it’s not confined to a local machine. So whatever Executor you use is agnostic to the backend. The executor is already at the point where the flow was obtained, and as far as they are concerned, they just update the states of the tasks by hitting the Prefect API of your

backend

. The LocalExecutor is still capable of making outside connections. You can hit other APIs. You can hit AWS resources. The local just means compute is running on that machine. So you can still use it to interact with Prefect Cloud and retrieve values from the KV Store because it’s pretty much the same as hitting an API. You just need to be authenticated on the Client. In fact, you can hit the Prefect KV Store outside of Prefect flows as well as long as you’re authenticated to the Prefect Client. Does that clear things up?