https://prefect.io logo
Title
l

Lon Nix

11/03/2021, 5:56 PM
In my flow I define this 
job_template
job_template = '''
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
        - name: flow
          volumeMounts:
            - name: ssh-key
              readOnly: true
              mountPath: "/root/.ssh"
      volumes:
        - name: ssh-key
          secret:
            secretName: prefect-ssh-key
            optional: false
            defaultMode: 0600
'''
but what shows up for the job is actually
volumeMounts:
    - mountPath: /root/.ssh
      name: ssh-key
      readOnly: true
  volumes:
  - name: ssh-key
    secret:
      defaultMode: 384
      optional: false
      secretName: prefect-ssh-key
It's not keeping the same permissions for 
defaultMode
  and I think that's why I'm getting an error about 
Failed to add the RSA host key for IP address '140.82.114.4' to the list of known hosts
 It is using the correct secret name though. Why would it not take the correct defaultMode?
I tried with and without
readOnly: true
and it didn't make a difference
k

Kevin Kho

11/03/2021, 6:06 PM
Just to understand better, you define this job template and then you add it to the KubernetesRun? Do you have anything on the agent side?
l

Lon Nix

11/03/2021, 6:09 PM
Yes, I define that template then do
flow.run_config = KubernetesRun(job_template=job_template)
k

Kevin Kho

11/03/2021, 6:11 PM
How do you start the agent?
l

Lon Nix

11/03/2021, 6:19 PM
It's got its own yaml that gets applied via kubectl. @Xinchi He can you weigh in more?
k

Kevin Kho

11/03/2021, 6:20 PM
I am just wondering if it’s possible the defaultMode is coming from the agent somehow
l

Lon Nix

11/03/2021, 6:22 PM
if it is its not something we're specifying. We're running a pretty standard agent configuration, just added some env vars for our system
x

Xinchi He

11/03/2021, 6:25 PM
Sure. it's the automatically generated yaml for the agent, the only thing we changed is API URL, label, env vars and image (we use prefect-python3.9:latest for the agent)
k

Kevin Kho

11/03/2021, 6:25 PM
I am looking at this . did you come across it?
But yours doesn’t seem like the default value? (My k8s knowledge is bad btw). What is your version?
l

Lon Nix

11/03/2021, 6:32 PM
sorry, my mistake here, apparently a defaultMode of 384 is actually setting permissions to 0600 which is what we want. idk why k8s doesn't report it the same way
still not sure why we can't add to known_hosts though
k

Kevin Kho

11/03/2021, 6:33 PM
I just saw that. 0600 is octal and 384 is decimal so your defaultMode is indeed working
I will ping someone on the team who worked on this. Dunno when they’ll be availabel though
l

Lon Nix

11/03/2021, 6:36 PM
it would be great to have more insight into what causes the error. I'm not sure if its while trying to do a git clone or something else, it's all speculation at this point. And there is no output as to what it is trying to clone if that is indeed the issue
k

Kevin Kho

11/03/2021, 6:38 PM
The best advice for debugging is to run the Docker image interactively and try running 
git clone <ssh repo url>
 in the interactive shell. I assume this doesn’t work for you?
l

Lon Nix

11/03/2021, 6:40 PM
I did get it working once doing that but I've been unable to get it to work when running in k8s. It always errors with
Failed to load and execute Flow's environment: HangupException("Failed to add the RSA host key for IP address '140.82.112.4' to the list of known hosts (/root/.ssh/known_hosts).\r\<mailto:ngit@github.com|ngit@github.com>: Permission denied (publickey).\r")