https://prefect.io logo
Title
t

Troy Sankey

04/29/2020, 5:45 PM
I have a basic misunderstanding about how the KubernetesJobEnvironment creates jobs. I keep trying to add items into my job_spec_file which gets passed into KubernetesJobEnvironment, but it doesn't seem like they're being picked up. Like, when I try to add spec.template.spec.serviceAccount and spec.template.spec.serviceAccountName, the containers that get create don't mount the token for the specified service account (they just use the
default
one instead). Also, more fundamentally, the prefect code is hard-coded to use the following string for the container args: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/environments/execution/k8s/job.py#L276
"python -c 'import prefect; prefect.Flow.load(prefect.context.flow_file_path).environment.run_flow()'"
but actually the job spec ends up using a different string!
$ kubectl --namespace=prefect get job prefect-job-e4b101a5 -o yaml | grep -A1 args
      - args:
        - prefect execute cloud-flow
I checked the version of prefect on my laptop (which i use to deploy the prefect flow) and the prefect agent image deployed to my cluster. It's all 0.10.x.
👀 1
j

josh

04/29/2020, 5:53 PM
Hey @Troy Sankey so a bit of clarification here: the 
prefect-job-xx
 that you’re looking at is the initial prefect job that the agent creates and not the job created by the K8sJobEnvironment. When the Agent finds a flow run it creates a job using the image you have set as your Flow’s 
Docker
 storage. That job acts as a sort of init container where your Flow is loaded, inspected, and executed. In this case it is looking at your flow’s K8sJobEnvironment and creating a new job to execute your flow on. Then the original prefect job completes and the new job that was created runs the flow.
i.e. there are two jobs, the first one running
prefect execute cloud-flow
to execute the flow’s environment and the second running the environment’s
run_flow()
function to actually run the flow
Are you seeing the job you set for the Flow’s environment being created at all?
t

Troy Sankey

04/29/2020, 6:21 PM
i'll look again and see if i can find it
no, all I see are prefect-job-xx jobs, and some other unrelated jobs, but none with the name that I have specified in the spec that I supplied to K8sJobEnvironment
j

josh

04/29/2020, 6:24 PM
Do you see anything in the logs of the prefect jobs that could indicate failure?
t

Troy Sankey

04/29/2020, 6:25 PM
yea, it clearly was running my code, since it threw an exception on a line in the middle of my task
j

josh

04/29/2020, 6:25 PM
Just checking, are you attaching the KubernetesJobEnvironment to your
flow.environment
?
t

Troy Sankey

04/29/2020, 6:26 PM
yep yep, specified during constructing the Flow. would it help if i pasted this whole flow file?
may a gist
j

josh

04/29/2020, 6:27 PM
Yeah any reproduceable example would be helpful 😄
t

Troy Sankey

04/29/2020, 6:30 PM
woah, while i was prepping the Gist, i think i noticed a typo:
with Flow("Test credentials", environment=environment, storage=storage) as flow:
    sf_credentials = VaultKVSecret(path="snowflake_pipeline_etl_loader", version=1,)
    test_connection(sf_credentials)

flow.environment = RemoteEnvironment(labels=["prod"])
flow.register(project_name="tsankey test project")
i think i'm overriding the environment on accident
j

josh

04/29/2020, 6:31 PM
Ah that would do it!
t

Troy Sankey

04/29/2020, 6:31 PM
welp, that gives me hope for the next test run 🙂
now the first job is failing to create the second job because we did not give any permissions to the default service account (annoyingly the default service account won't go away even if we delete it. it just keeps reappearing and we don't know why)
as a workaround we'll permit prefect:default service account to create jobs in the prefect namespace, but I wonder if there's a way to configure the first job to use a specific service account? (we want them to use
prefect:prefect
)
according to this function, it doesn't really seem like there are any hooks to add in a serviceAccount: https://github.com/PrefectHQ/prefect/blob/970c8527af69ca126879bdf76cb6bb079054ec06/src/prefect/agent/kubernetes/agent.py#L100
j

josh

04/29/2020, 6:51 PM
Yeah currently there isn’t a way to set a default service account for those jobs but there should be! Would you mind opening an issue for that?
t

Troy Sankey

04/29/2020, 7:13 PM
sure! i've worked around it by adding a role binding for now.
👍 1