Hi We have a set of workflows that we are in process of pref

Question

Hi We have a set of workflows that we are in process of prefectifying but have stumbled on a problem We are processing weather data and the preferred python libraries for handling meteorological data NetCDF expect files to be accessible on a mounted filesystsem Our preferred option is to run flows on kubernetes but it is not clear to us how to mount a volume on the pods that execute the flows All tips greatly appreciated

Chris White · Answer

Hi < Espen Overbye> if you are using a backend API then the `KubernetesJobEnvironment` allows you to specify a job spec for the flow which can include mounted volumes

Espen Overbye · Answer

Hi < Chris White> I m eager to try out with now merged

Espen Overbye · Answer

Hi < Chris White> after having tested this a fair bit more I m still stuck The doc is not in sync with what I m experiencing Registering the flow works fine but the job is still executed with the job spec yaml that seems to be default on the agent and not the one I ve defined It could be that I misunderstand how agent kubernetes is related to execution k8s

Espen Overbye · Answer

Having the code image in a private repo makes it sort of hard to create a full running example

josh · Answer

< Espen Overbye> could you post an example of your flow environment and how you are creating it

Espen Overbye · Answer

``` flows my flow py from prefect import task Flow from prefect environments storage import GitHub Docker from prefect environments import LocalEnvironment KubernetesJobEnvironment import pathlib task def get data return 1 2 3 4 5 task def print data data print data with Flow test as flow data = get data print data data flow storage = GitHub repo= airmine ai prefect name of repo path= flows my flow py location of flow file in repo secrets= GITHUB ACCESS TOKEN name of personal access token secret flow environment = KubernetesJobEnvironment job spec file= job yaml metadata= image <http airmineacrprod azurecr io griddr prefect 2020 08 11|airmineacrprod azurecr io griddr prefect 2020 08 11> ```

Espen Overbye · Answer

```apiVersion batch v1 kind Job metadata name prefect job UUID labels app prefect job UUID identifier UUID spec template metadata labels app prefect job UUID identifier UUID spec containers name flow image <http airmineacrprod azurecr io griddr prefect 2020 08 10|airmineacrprod azurecr io griddr prefect 2020 08 10> imagePullPolicy IfNotPresent restartPolicy Never volumeMounts name azure mountPath mnt data volumes name azure azureFile secretName griddeddata shareName gridded data readOnly false```

Espen Overbye · Answer

```Agent deployed with prefect agent install```

Espen Overbye · Answer

cli and griddr prefect updated to prefect 0 13 1

josh · Answer

In your image ` <http airmineacrprod azurecr io griddr prefect 2020 08 11|airmineacrprod azurecr io griddr prefect 2020 08 11> ` is there a file called `job yaml` Generally with storage options in the past the job spec file is loaded at registration time and shipped with the flow pickle to wherever you store it However I m not 100 sure how this plays with file based storage like github because the flow is not initialized until it goes to run to which it requires that the job spec yaml is present there

Espen Overbye · Answer

No my understanding was that <https github com PrefectHQ prefect pull 2950> was solving that issue in other words making sure the job spec file was pulled from github

Espen Overbye · Answer

and from the pod job logs it seems like prefect is looking for job spec yaml though I specified job yaml

Espen Overbye · Answer

PS C Users espen source repos airmine ai griddr gt kc logs prefect job 7fa422e0 5tvkq Errno 2 No such file or directory job spec yaml Traceback most recent call last File usr local bin prefect line 8 in lt module gt sys exit cli File usr local lib python3 8 dist packages click core py line 829 in call return self main args kwargs File usr local lib python3 8 dist packages click core py line 782 in main rv = self invoke ctx File usr local lib python3 8 dist packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 8 dist packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File usr local lib python3 8 dist packages click core py line 1066 in invoke return ctx invoke self callback ctx params File usr local lib python3 8 dist packages click core py line 610 in invoke return callback args kwargs File usr local lib python3 8 dist packages prefect cli execute py line 46 in cloud flow return execute flow run File usr local lib python3 8 dist packages prefect cli execute py line 93 in execute flow run raise exc File usr local lib python3 8 dist packages prefect cli execute py line 83 in execute flow run flow = storage get flow storage flows flow data name File usr local lib python3 8 dist packages prefect environments storage github py line 89 in get flow return extract flow from file file contents=decoded contents File usr local lib python3 8 dist packages prefect utilities storage py line 74 in extract flow from file exec contents exec vals File lt string gt line 25 in lt module gt File usr local lib python3 8 dist packages prefect environments execution k8s job py line 87 in init self job spec = self load spec from file File usr local lib python3 8 dist packages prefect environments execution k8s job py line 337 in load spec from file with open self job spec file as job spec file FileNotFoundError Errno 2 No such file or directory job spec yaml

josh · Answer

Hmm is there a chance you haven t updated the flow file in github and it still says `job spec yaml` That error is showing it initializing your environment and attempting to load the yaml file it has listed under `self job spec file` and that value is initialized as ```self job spec file = os path abspath job spec file if job spec file```

josh · Answer

So `job spec yaml` is being passed into the environment somewhere

Espen Overbye · Answer

Maybe the problem is that I have assumed that the kubernetes agent unpickles the flow and the job spec but it could be that there is an extra step here Agent reads the flow spec but does not read the flow from github Instead it runs the job at <https github com PrefectHQ prefect blob master src prefect agent kubernetes job spec yaml> This is inline with the job definition I see created in Kubernetes This will run a new pod pulling the flow spec from github Because I m running a custom image without the job spec yaml or ideally job yaml this step fails

Espen Overbye · Answer

The kubernetes agent and the kubernetesjobenvironment is confusing

josh · Answer

Due to the separation of user data and flow metadata there is an intermediate step happening here The agent creates an initial prefect job where the flow s storage is used to grab the flow from github Then the flow is loaded up and the KubernetesJobEnvironment is executed to create the job with your custom specs Yeah this pattern is confusing and that s why we are planning to deprecate it in the future for something much cleaner slightly smiling face <https github com PrefectHQ prefect issues 2928>

Espen Overbye · Answer

That sounds good any idea on timeline for this Might fall back to skipping github and using docker as storage for now

Espen Overbye · Answer

And thanks for helping out