In my flow I define this 

job_template

job_template = '''
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
        - name: flow
          volumeMounts:
            - name: ssh-key
              readOnly: true
              mountPath: "/root/.ssh"
      volumes:
        - name: ssh-key
          secret:
            secretName: prefect-ssh-key
            optional: false
            defaultMode: 0600
'''

but what shows up for the job is actually

volumeMounts:
    - mountPath: /root/.ssh
      name: ssh-key
      readOnly: true
  volumes:
  - name: ssh-key
    secret:
      defaultMode: 384
      optional: false
      secretName: prefect-ssh-key

It's not keeping the same permissions for 

defaultMode

  and I think that's why I'm getting an error about 

Failed to add the RSA host key for IP address '140.82.114.4' to the list of known hosts

 It is using the correct secret name though. Why would it not take the correct defaultMode?

Just to understand better, you define this job template and then you add it to the KubernetesRun? Do you have anything on the agent side?

Yes, I define that template then do

flow.run_config = KubernetesRun(job_template=job_template)

How do you start the agent?

It's got its own yaml that gets applied via kubectl. @Xinchi He can you weigh in more?

I am just wondering if it’s possible the defaultMode is coming from the agent somehow

if it is its not something we're specifying. We're running a pretty standard agent configuration, just added some env vars for our system

Sure. it's the automatically generated yaml for the agent, the only thing we changed is API URL, label, env vars and image (we use prefect-python3.9:latest for the agent)

I am looking at this . did you come across it?

sorry, my mistake here, apparently a defaultMode of 384 is actually setting permissions to 0600 which is what we want. idk why k8s doesn't report it the same way

I just saw that. 0600 is octal and 384 is decimal so your defaultMode is indeed working

it would be great to have more insight into what causes the error. I'm not sure if its while trying to do a git clone or something else, it's all speculation at this point. And there is no output as to what it is trying to clone if that is indeed the issue

The best advice for debugging is to run the Docker image interactively and try running 

git clone <ssh repo url>

 in the interactive shell. I assume this doesn’t work for you?

I did get it working once doing that but I've been unable to get it to work when running in k8s. It always errors with

Failed to load and execute Flow's environment: HangupException("Failed to add the RSA host key for IP address '140.82.112.4' to the list of known hosts (/root/.ssh/known_hosts).\r\<mailto:ngit@github.com|ngit@github.com>: Permission denied (publickey).\r")