Tom Klein
04/04/2022, 1:16 PMKevin Kho
Tom Klein
04/04/2022, 1:37 PMKevin Kho
flow.run_config = RunConfig(..., env={...})
but this will be the same values across all flow runs
I think you can register with this and then use the UI to override for the ad-hoc backfillsAnna Geller
Tom Klein
04/04/2022, 1:42 PMAnna Geller
i’m talking about altering them from run to run (but still having most of them “persisted”)It seems like you are looking for some external storage solution like Redis - it would allow you to do that. You shouldn't really use Prefect as a parameter store; Prefect is mostly about orchestration and execution
our flow actually runs (as one of its steps) a Kubernetes job of a specific docker image, and its that internal job that needs to have the env vars, not the flowmaybe Kubernetes secrets is the right approach here?
Tom Klein
04/04/2022, 2:14 PMKevin Kho
Anna Geller
because it’s orchestrated by Prefect i’m not sure what the correct approach would beI don't think Prefect puts any restrictions in that regard. You can totally still use parameter store or secrets manager for that, and this would be actually quite useful because it makes your code easy to migrate to Prefect 2.0
Tom Klein
04/04/2022, 2:53 PMAnna Geller
Tom Klein
04/05/2022, 12:26 PMAnna Geller
Tom Klein
04/05/2022, 12:34 PMRunNamespacedJob
task
Basically all we’re trying to do is use the flow to orchestrate several k8s jobs - each having their own image… are we going about this the wrong way?
e.g. we have a process that :
• pulls data from our DWH
• runs some NodeJS code to do stuff with that data (e.g. scrape websites)
• runs a python-based ML model on the data
• exports the results to S3
and up until now we ran it manually which is error prone so we’re trying to migrate it to PrefectAnna Geller
Tom Klein
04/05/2022, 12:47 PMRunNamespacedJob
command as a way to execute a program [encapsulated in a Docker image] that was initially designed to run as a standalone process rather than as a step in an orchestrated DAG)Anna Geller