Domantas

    Domantas

    1 year ago
    Hello guys, I would like to ask if anyone have an example how to mount CephFS with prefect flow? For now I'm trying to use
    job_template_path
    parameter in
    KubernetesRun
    by passing yaml file which looks like this:
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: prefect-job
      labels: {}
    spec:
      template:
        metadata:
          labels: {}
        spec:
          containers:
            - name: flow
              image: test-image:1.2
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: ["prefect execute flow-run"]
              env:
                - name: PREFECT__CONTEXT__SECRETS__database_psw
                  value: veryStrongPasswrod
              volumeMounts:
                - name: storage
                  mountPath: /tnt/test/domantas_test
          restartPolicy: Never
          imagePullSecrets:
          - name: registry-secret
          volumes:
            - name: storage
              cephfs:
                monitors:
                    - 59.11.129.131
                    - 59.11.129.132
                    - 59.11.129.133
                    - 59.11.129.134
                    - 59.11.129.135
                path: /test
                user: storage
                secretRef:
                    name: ceph-secret
    However, when I try to execute in prefect server UI, it seems the flow get stuck with "Submitted for execution" status with no useful logs. I took job_template example from here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/job_spec.yaml and additionally added cephFS necessary parameters.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Domantas! I think your issue might not be CephFS. Could you set your agent log level to debug and also run the flow on debug. This can now be done through the UI.
    Just go to Run -> Advanced Configuration and set to DEBUG
    And just a friendly reminder to move the yaml into the thread if possible.
    Domantas

    Domantas

    1 year ago
    Strangely, I'm unable to see Logging level option in the Run configuration settings:
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh then on the agent side just set log level to debug, and --show-flow-logs
    I think this thread might be helpful for you
    Domantas

    Domantas

    1 year ago
    Is this a bad thing if I'm loading yaml file from my local file system into the agent?
    flow.run_config = KubernetesRun(
        cpu_request=2,
        memory_request="2Gi",
        image="test-image:1.2",
        image_pull_secrets=["registry-secret"],
        job_template_path="/home/Projects/prefect/job_template.yaml",
    )
    Maybe this is the root of the problem that agent is unable to pick latest job template when kubernete container is created? Just for testing, I uploaded job_template.yaml file into s3 storage, and trying to fetch it from there. However, prefect throws an error:
    Unable to locate credentials
    . If this is the root of the issue, how to pass aws credentials into KubernetesRun? (I checked https://docs.prefect.io/api/latest/run_configs.html#kubernetesrun and could not find any example of how to to that)
    Kevin Kho

    Kevin Kho

    1 year ago
    It might be. Here you can pass the environment variables through KubernetesRun through the
    env
    keyword. Here is the format for aws credentials
    Domantas

    Domantas

    1 year ago
    Is this possible to declare aws endpoint address in the default prefect secret form? Current configuration looks like this:
    flow.run_config = KubernetesRun(
        cpu_request=2,
        memory_request="2Gi",
        image="test-image:1.2",
        image_pull_secrets=["registry-secret"],
        job_template_path="<s3://prefect/job_template.yaml>",
        env={
            "PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS": {"ACCESS_KEY": aws_access_key_id, "SECRET_ACCESS_KEY": aws_secret_access_key}
        }
    )
    I still get an error:
    botocore.exceptions.NoCredentialsError: Unable to locate credentials
    Default aws credentials are properly declared in
    ~/.aws/credentials
    file Also I'm using S3Result method, so there I setup credentials through
    client_options
    parameters which seems to work properly.
    Kevin Kho

    Kevin Kho

    1 year ago
    Deleted previous msg. Might not be applicable. Let me dig a bit. Sorry for the delayed responses.
    The local job template path will work. The issue is not being unable to pull it
    Domantas

    Domantas

    1 year ago
    Thank you @Kevin Kho, there are a lot of concepts that I'm not fully understand, so these references/documentation links helps a lot! I think the first thing that I should fix is logging problem, since it is not clear why code get stuck in the server UI. "Oh then on the agent side just set log level to debug, and --show-flow-logs" - if you don't mind, I would like to ask follow up question according to this answer: How to set up
    --show-flow-logs
    on
    Kubernete agent
    ? The only available option related with this answer is setting debug log level(https://docs.prefect.io/api/latest/cli/agent.html#kubernetes-start). However, setting debug log option is still not enough, because there are no logs why do code state get stuck into
    Submitted
    state(https://docs.prefect.io/orchestration/faq/debug.html#my-flow-is-stuck-in-a-submitted-state).
    Got some updates, debugging logs seems to be working, now there are more logs than usual. These errors occurs when I'm trying to load
    job_template yaml file
    into the KuberneteRun, so I think something is wrong with the content of that yaml file. The bad thing, that I'm still not able to detect from the logs which part of yaml configuration is causing me troubles 🤔
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh you're right. --show-flow-logs is not available for kubernetes agent.
    Domantas

    Domantas

    1 year ago
    Hello @Kevin Kho. Problem solved and it was related with the yaml template - env part. Thank you for your help!