Hello Prefect! I have a question about variables defined in the config.toml file. Can you confirm that when registering a flow any variables defined in config.toml are “uploaded” to Prefect Cloud as part of the registration process? For example when instantiating classes with config.toml configuration: refresh_mv = PostgresExecute(name=“Refresh MV”, host=prefect.config.postgres.host, db_name=prefect.config.postgres.db_name, user=prefect.config.postgres.user) I expected to need to define those configuration variables as environment variables in the execution rutime but did not. Thanks in advance! rb

Hi @Robert Bastian - the docs on config should give some background here. We don't "upload" much to Prefect Cloud - everything is run on your infrastructure - you can see more about data handling here.

@Jenny Thanks, but what’s confusing me is that my flow running in the cloud had configuration I didn’t provide it (or didn’t know I did!). What I’m assuming must be happening is that my agent is currently running on my laptop and must also have access to my local config.toml and must be providing the prefect.config.* values to my flows. Does that make sense?

Yes - if the agent is running as a local agent on your laptop it would have access to your config.toml on your laptop.

@Jenny I’m running an ecs agent on my laptop but I’m guessing the result is the same.

Hey - @Robert Bastian - sorry for the delay - I'm double checking this one with the team!

@Jenny The variables are defined only in my config.toml file on my laptop. I’m pretty sure that when I register a flow using s3 storage they are copied into the cloud-pickled flow. However, I tried to write a tool to “look at” the cloud-pickled flow but couldn’t get it to work. Here is how the variables are assigned in my flow currently:

# Instantiate PG Tasks
refresh_mv = PostgresExecute(name="Refresh MV", host=prefect.config.postgres.host, db_name=prefect.config.postgres.db_name, user=prefect.config.postgres.user)
count_mv = PostgresFetch(name="Count MV", host=prefect.config.postgres.host, db_name=prefect.config.postgres.db_name, user=prefect.config.postgres.user)
group_by_mv = PostgresFetch(name="GroupBy MV", host=prefect.config.postgres.host, db_name=prefect.config.postgres.db_name, user=prefect.config.postgres.user)

Hi Robert, In this case your config variables aren't uploaded to cloud (we never store your secret stuff in our DB). Rather, they're being stored along with your flow when you register your flow. This only happens for pickle-based flow storage https://docs.prefect.io/orchestration/flow_config/storage.html#pickle-vs-script-based-storage. When you register your flow, your flow script is executed to build the flow. Your config variables are loaded at this point, and stored in your tasks. The flow is then serialized using pickle (taking the config variables along with it). At runtime the flow is unpickled and whatever configuration was used at build time is still used. If you used script-based storage your flow would be rebuilt on every execution, and the config would be required in the runtime environment. Does that make sense?

@Jim Crist-Harif Yes, it does! TY!