Hello guys, I would like to ask if anyone have an example how to mount CephFS with prefect flow? For now I'm trying to use

job_template_path

parameter in

KubernetesRun

by passing yaml file which looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: test-image:1.2
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh", "-c"]
          args: ["prefect execute flow-run"]
          env:
            - name: PREFECT__CONTEXT__SECRETS__database_psw
              value: veryStrongPasswrod
          volumeMounts:
            - name: storage
              mountPath: /tnt/test/domantas_test
      restartPolicy: Never
      imagePullSecrets:
      - name: registry-secret
      volumes:
        - name: storage
          cephfs:
            monitors:
                - 59.11.129.131
                - 59.11.129.132
                - 59.11.129.133
                - 59.11.129.134
                - 59.11.129.135
            path: /test
            user: storage
            secretRef:
                name: ceph-secret

However, when I try to execute in prefect server UI, it seems the flow get stuck with "Submitted for execution" status with no useful logs. I took job_template example from here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/job_spec.yaml and additionally added cephFS necessary parameters.

Hi @Domantas! I think your issue might not be CephFS. Could you set your agent log level to debug and also run the flow on debug. This can now be done through the UI.

Strangely, I'm unable to see Logging level option in the Run configuration settings:

Oh then on the agent side just set log level to debug, and --show-flow-logs

Is this a bad thing if I'm loading yaml file from my local file system into the agent?

flow.run_config = KubernetesRun(
    cpu_request=2,
    memory_request="2Gi",
    image="test-image:1.2",
    image_pull_secrets=["registry-secret"],
    job_template_path="/home/Projects/prefect/job_template.yaml",
)

Maybe this is the root of the problem that agent is unable to pick latest job template when kubernete container is created? Just for testing, I uploaded job_template.yaml file into s3 storage, and trying to fetch it from there. However, prefect throws an error:

Unable to locate credentials

. If this is the root of the issue, how to pass aws credentials into KubernetesRun? (I checked https://docs.prefect.io/api/latest/run_configs.html#kubernetesrun and could not find any example of how to to that)

It might be. Here you can pass the environment variables through KubernetesRun through the

env

keyword. Here is the format for aws credentials

Is this possible to declare aws endpoint address in the default prefect secret form? Current configuration looks like this:

flow.run_config = KubernetesRun(
    cpu_request=2,
    memory_request="2Gi",
    image="test-image:1.2",
    image_pull_secrets=["registry-secret"],
    job_template_path="<s3://prefect/job_template.yaml>",
    env={
        "PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS": {"ACCESS_KEY": aws_access_key_id, "SECRET_ACCESS_KEY": aws_secret_access_key}
    }
)

I still get an error:

botocore.exceptions.NoCredentialsError: Unable to locate credentials

Default aws credentials are properly declared in

~/.aws/credentials

file Also I'm using S3Result method, so there I setup credentials through

client_options

parameters which seems to work properly.

Deleted previous msg. Might not be applicable. Let me dig a bit. Sorry for the delayed responses.

Thank you @Kevin Kho, there are a lot of concepts that I'm not fully understand, so these references/documentation links helps a lot! I think the first thing that I should fix is logging problem, since it is not clear why code get stuck in the server UI. "Oh then on the agent side just set log level to debug, and --show-flow-logs" - if you don't mind, I would like to ask follow up question according to this answer: How to set up

--show-flow-logs

on

Kubernete agent

? The only available option related with this answer is setting debug log level(https://docs.prefect.io/api/latest/cli/agent.html#kubernetes-start). However, setting debug log option is still not enough, because there are no logs why do code state get stuck into

Submitted

state(https://docs.prefect.io/orchestration/faq/debug.html#my-flow-is-stuck-in-a-submitted-state).

Oh you're right. --show-flow-logs is not available for kubernetes agent.

Hello @Kevin Kho. Problem solved and it was related with the yaml template - env part. Thank you for your help!