t

    Troy Sankey

    2 years ago
    I have a basic misunderstanding about how the KubernetesJobEnvironment creates jobs. I keep trying to add items into my job_spec_file which gets passed into KubernetesJobEnvironment, but it doesn't seem like they're being picked up. Like, when I try to add spec.template.spec.serviceAccount and spec.template.spec.serviceAccountName, the containers that get create don't mount the token for the specified service account (they just use the
    default
    one instead). Also, more fundamentally, the prefect code is hard-coded to use the following string for the container args: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/k8s/job.py#L276
    "python -c 'import prefect; prefect.Flow.load(prefect.context.flow_file_path).environment.run_flow()'"
    but actually the job spec ends up using a different string!
    $ kubectl --namespace=prefect get job prefect-job-e4b101a5 -o yaml | grep -A1 args
          - args:
            - prefect execute cloud-flow
    I checked the version of prefect on my laptop (which i use to deploy the prefect flow) and the prefect agent image deployed to my cluster. It's all 0.10.x.
    j

    josh

    2 years ago
    Hey @Troy Sankey so a bit of clarification here: the 
    prefect-job-xx
     that you’re looking at is the initial prefect job that the agent creates and not the job created by the K8sJobEnvironment. When the Agent finds a flow run it creates a job using the image you have set as your Flow’s 
    Docker
     storage. That job acts as a sort of init container where your Flow is loaded, inspected, and executed. In this case it is looking at your flow’s K8sJobEnvironment and creating a new job to execute your flow on. Then the original prefect job completes and the new job that was created runs the flow.
    i.e. there are two jobs, the first one running
    prefect execute cloud-flow
    to execute the flow’s environment and the second running the environment’s
    run_flow()
    function to actually run the flow
    Are you seeing the job you set for the Flow’s environment being created at all?
    t

    Troy Sankey

    2 years ago
    i'll look again and see if i can find it
    no, all I see are prefect-job-xx jobs, and some other unrelated jobs, but none with the name that I have specified in the spec that I supplied to K8sJobEnvironment
    j

    josh

    2 years ago
    Do you see anything in the logs of the prefect jobs that could indicate failure?
    t

    Troy Sankey

    2 years ago
    yea, it clearly was running my code, since it threw an exception on a line in the middle of my task
    j

    josh

    2 years ago
    Just checking, are you attaching the KubernetesJobEnvironment to your
    flow.environment
    ?
    t

    Troy Sankey

    2 years ago
    yep yep, specified during constructing the Flow. would it help if i pasted this whole flow file?
    may a gist
    j

    josh

    2 years ago
    Yeah any reproduceable example would be helpful 😄
    t

    Troy Sankey

    2 years ago
    woah, while i was prepping the Gist, i think i noticed a typo:
    with Flow("Test credentials", environment=environment, storage=storage) as flow:
        sf_credentials = VaultKVSecret(path="snowflake_pipeline_etl_loader", version=1,)
        test_connection(sf_credentials)
    
    flow.environment = RemoteEnvironment(labels=["prod"])
    flow.register(project_name="tsankey test project")
    i think i'm overriding the environment on accident
    j

    josh

    2 years ago
    Ah that would do it!
    t

    Troy Sankey

    2 years ago
    welp, that gives me hope for the next test run 🙂
    now the first job is failing to create the second job because we did not give any permissions to the default service account (annoyingly the default service account won't go away even if we delete it. it just keeps reappearing and we don't know why)
    as a workaround we'll permit prefect:default service account to create jobs in the prefect namespace, but I wonder if there's a way to configure the first job to use a specific service account? (we want them to use
    prefect:prefect
    )
    according to this function, it doesn't really seem like there are any hooks to add in a serviceAccount: https://github.com/PrefectHQ/prefect/blob/970c8527af69ca126879bdf76cb6bb079054ec06/src/prefect/agent/kubernetes/agent.py#L100
    j

    josh

    2 years ago
    Yeah currently there isn’t a way to set a default service account for those jobs but there should be! Would you mind opening an issue for that?
    t

    Troy Sankey

    2 years ago
    sure! i've worked around it by adding a role binding for now.