Hi,
i have a flow with docker storage in ECR and a...
# ask-communityn
Nadav
09/02/2021, 6:05 PMHi,
i have a flow with docker storage in ECR and a KubernetesRun with image_pull_policy Always , i have an agent running on my EKS cluster, for some reason when i change the flow code and builds a new image and run the flow from cloud, the flow runs with the old image, i validated this when i change the image tag then the new code is used, any clues how to solve this?
k
Kevin Kho
09/02/2021, 6:07 PM
Can I see how you set up KubernetesRun?
n
Nadav
09/02/2021, 6:10 PMflow.run_config = KubernetesRun(
image=‘*************.dkr.ecr.us-west-2.amazonaws.com/semantic-layer:copy_semantic_views_to_snowflake2’,
image_pull_policy=“Always”
)
k
Kevin Kho
09/02/2021, 6:21 PM
This looks good to me. Will ask the team for any ideas
Kevin Kho
09/02/2021, 6:48 PM
Hey, would you be able to check if the `image_pull_policy`is applied to the job that the agent created?
b
Brad I
09/03/2021, 5:23 AMI may have run into the same issue in GKE: https://github.com/PrefectHQ/prefect/issues/4939
k
Kevin Kho
09/03/2021, 5:39 AM
Ok I’ll try this tom
z
Zach Angell
09/03/2021, 2:53 PM
@Brad I a couple questions to help in debugging
• are you defining anything on your Kubernetes Agent? (e.g. a job template?)
• how exactly are you running the flow? (UI, CLI, via a Schedule)
• if running the flow in the UI, how exactly are you creating the flow run?
b
Brad I
09/03/2021, 3:08 PMHi @Zach Angell
• We are deploying the standard agent yaml config from the command:
prefect agent kubernetes installIMAGE_PULL_POLICYIfNotPresentcreate_flow_runKubernetesRunimage_pull_policyz
Zach Angell
09/03/2021, 3:17 PM
Thank you for the detail here!
For the UI, is image pull policy respected if you just click "Quick Run"?
For the API, could you share the
create_flow_runZach Angell
09/03/2021, 4:00 PM
I haven't had a chance to test, but for the UI I suspect we need to add an "Image Pull Policy" field to the "Run" tab. If you edit the flow's run config while creating a flow run, it will overwrite the run config specified at registration.
For the API, it's possible the Flow Group's run config is overriding the Flow's run config. Although this is not a common scenario.
If you specify the full
run_configcreate_flow_runimage_pull_policyb
Brad I
09/03/2021, 7:48 PMI can try that from the API, we currently don’t override anything other than the input parameters when creating a run since the client doesn’t know the correct backend environment variables to use. Our query looks like:
Copy code
const query = gql`
mutation ($input: create_flow_run_input!) {
create_flow_run(input: $input) {
id
}
}
`;
const variables = {
input,
}; Copy code
parameters = {
some_var: 42,
uuid: fileId,
}Brad I
09/03/2021, 7:50 PMIn general, we don’t really want our users to have to configure every run_config parameter and should just use the default that the flow was registered with (just like all the default env vars we set).
z
Zach Angell
09/07/2021, 3:58 PM
In general, we don’t really want our users to have to configure every run_config parameter and should just use the default that the flow was registered withUnderstood and 100% agree that is how the system should work. For the UI, we need to add a new field. I've opened an issue here https://github.com/PrefectHQ/ui/issues/1040. For the API, I can do some further testing. Could you DM me the ID of your Flow and Flow Group? Both can be found in the "Flow" page -> "Overview" tab -> "Details" option on the "Overview" tile"
Zach Angell
09/08/2021, 10:11 PM
@Brad I I did some testing against the API and I'm not able to reproduce the issue you're seeing. Unless
run_configcreate_flow_runrun_configb
Brad I
09/10/2021, 12:40 PMThanks @Zach Angell, we’ve been doing a lot of testing just from the UI, so I’ll double check from the API today.
👍 1
6 Views