Howdy, when an Agent triggers a run of a registered flow, is there some standard way to ask Agent to pass some arguments into the flow? e.g. Agent would do:

registerd_flow.run(**runtime_args_passed_by_agent)

Or is this disallowed by design? (registered flow shall be self-sufficient for reproducibility) Context: we have some flow that depends on some value that changes quite often (e.g. dependency package version number). We don’t want to register a new flow each time we update the version number; rather have one flow and have the agent run the flow expecting the package version information will be provided to it at run time. We thought about using env var, but not sure if it’s the best way…

Hi @dh! Prefect currently doesn't offer such functionality. It's certainly on our radar though and we'll be looking to provide something in the future. For now, the best thing would probably be to use some NoSQL database (like Firestore) and have your flow pull the values from there.

Hi @Kevin Kho thank you for sharing the advice. Do you have some reference example I can look into for the pattern you mentioned? Also --- alternatively I am thinking whether it would make sense to register as many flows as possible (with dynamic info such as number of gpus to be used for execution of some tasks), but keep them under some sensible version group. Questions: 1. Can version group be manually specified to something that can be different from flow_name? (e.g. version group: ml_job_flow, flow_name: ml_job_flow_k80x3 2. Is there a standard way to configure Agent to execute an arbitrary flow under a specific version group (not just the latest). Last year when I chat, I think the answer was no. I wonder if it still holds. 3. If answer is no above, is there a way for us to specific a logical group over multiple flows and execute any particular flow within the group? Even if there’s no standard way, any creative interim solution would be appreciated.

I dont have but I only bring up Firestore because it’s easy. I wouldn’t recommend it if you’re not on Google Cloud. Any NoSQL DB where you can persist will work. The answer to two is No because we don’t keep track of previous code. We only hold metadata.

Thank you for the advice. We’re not on GCP, so we will look into that direction also.

The answer to two is No because we don’t keep track of previous code. We only hold metadata.

Yes, I understand this is by design and that the flow code (the pickle-serialized flow object) gets stored on FlowStorage which users control not the server. Hence, I presume only metadata is saved even for “the latest” code? Please let me know if my mental model is wrong on this…

Are Parameters insufficient for your use case? You could put those specs as Parameters right?

Could you help me understand how the values for

Parameters

should be specified when Agent runs a registered flow pulled from FlowStorage (not users running the flow interactively in which case I know we can pass cli arguments with

flow.run(**cli_args)

? It would be lovely if Agent can do

flow.run(**my_dynamic_args)

. where the args can be pulled from somewhere (cli, database, …)

Yes on the metadata mental model. We have metadata for the previous versions but we can’t guarantee that the user will have the appropriate code (module may have changed). On the `flow.run()`dynamic args, you can pull them from a database and then feed it into the Flow as Parameters

@Kevin Kho thank you so much for sharing the advice. definitely will try this out!