prefect agent kubernetes install -k API_KEY | kubectl apply --namespace=my-namespace -f -

this command successfully creates the deployment and referrs to default prefect image so the question is how can i use my code with it?

Hey, have you seen this section of the docs? It will show you how to register a flow and run it. on that agent.

locally i have a code file that contains prefect flows and scheduler and my function so how can i reffer those with above deployment

The agent polls Prefect Cloud every 10 seconds for flows to run. It finds scheduled flows and then executes them, So you need to register the flow and schedule. A quick run also created a schedule.

@Aqib Fayyaz you need to add

KubernetesRun

run configuration to your flow. Here is one example: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run.py - it also depends on your flow storage. Do you use GCS as a Storage?

i am using Github storage

nice, then this example may work well for you

i have deployed the prefect-agent using the command prefect agent kubernetes install --rbac --key API_KEY | kubectl apply --namespace=YOUR_NAMESPACE -f - and it is running. After that i created the prefect file containing the code for now i am using the dumy example that you shared with me from prefect import Flow, task from prefect.storage import GitHub from prefect.run_configs import KubernetesRun FLOW_NAME = "github_kubernetes_run" STORAGE = GitHub( repo="aqibfayyaz/Work", path=f"flows/{FLOW_NAME}.py", access_token_secret="My Token", # required with private repositories ) @task(log_stdout=True) def hello_world(): text = f"hello from {FLOW_NAME}" print(text) return text with Flow( FLOW_NAME, storage=STORAGE, run_config=KubernetesRun(labels=["k8s"],), ) as flow: hw = hello_world() flow.register(project_name="tutorial") and i ran the command python3 mycodefile.py now i can see in prefect cloud ui kuberetes agent is there but when i run the task using using quickstart its just a yellow bar saying scheduled

any time you see that a flow gets stuck in a Scheduled state, the most likely culprit in most use cases is labels mismatch between a flow and agent. Note that in this example I defined a label “k8s” in the flow, because my KubernetesAgent also had this label assigned when starting the agent.

run_config=KubernetesRun(labels=["k8s"],)

I assume that you did not assign any labels by following the default configuration, so you can use KubernetesRun() with no labels:

run_config=KubernetesRun()

ok

You need to put a Secret with Prefect Cloud and then you provide the Secret name to your Github Storage. Not the value of the token.

Great thank you!

@Aqib Fayyaz is ghp_**** your Personal Access Token? if so, please remove it from the message for security reasons -> if it’s a random string, all good. Just checking for your security.

no its not

@Aqib Fayyaz in the same flow I shared before, I used a secret, as Kevin mentioned on this line.

ok

@Aqib Fayyaz you don’t need to explicitly grab it with PrefectSecret. As long as you set your GITHUB_ACCESS_TOKEN secret in cloud, you only need to pass is as reference, i.e.

GitHub(
    repo="your_user_name/your_repo",
    path="your/path/flow.py",
    access_token_secret="GITHUB_ACCESS_TOKEN"
)

Great thank you!

You can do that from the UI: https://cloud.prefect.io/team/members

ok Great

Git storage supports ssh I think. I think in general if you require ssh, you can’t use personal access token but I’m not 100% sure.

@Aqib Fayyaz Kevin is right. You’ll need to ensure that SSH client is installed and your SSH keys are configured in the agent’s env. There is even a designated documentation page for Git + SSH: https://docs.prefect.io/orchestration/flow_config/storage.html#ssh-git-storage

In case of github storage my repo is OpenAIMP/ADM and i have branch prefect_beta which contains flow for prefect. prefect_beta -> flow/flow.py and my code is STORAGE = GitHub( repo="OpenAIMP/ADM", path=f"prefect_beta/flow/{FLOW_NAME}.py", access_token_secret="OPENAIMP", # required with private repositories ) but every time flow fails with error File 'flow/flow.py' not found in repo 'OpenAIMP/ADM', ref 'master' i can see from error that it looks for flow in master but in my case it is in prefect_beta branch so how can i change it to refer to prefect_beta instead of master?

Git storage takes a branch name. Docs. For Github storage, I think the docstring of the

ref

parameter might help here. Docs

i have kubernetes agent working, flow on github storage is also working and it is successfully being executed by prefect cloud, now i just want to shift my account to another one how can i do that?

@Aqib Fayyaz I’m afraid this is not something we can offer at the moment. I believe the easiest way would be to generate a new API key in your new account, change the key on your agent and on the machine from which you register your flows, and then re-register your flows. But could you elaborate on what do you mean by shifting your work from one account to the other? Perhaps I misunderstood the request.

no you got it right. i was checking how can i shift from one gmail account to another on prefect cloud

@Aqib Fayyaz if all you want to change is the owner, what you could do as an alternative: • you can invite the other user (your second account) to the team you want to keep • then, make it an admin, • then you could either keep or even remove the first user

Great got it.

I think if you go to the

auth.toml

or

config.toml

in the

.prefect

folder, you can take out the key manually. Maybe you can also do

prefect auth logout

hi i have github storage working, kubernetes agent working and that dummy examples of hello to flow also work. now i need to send data to postgres and for that i have file sendDataToPostgres.py which has function sendDataToOfflineStore which does all the things and i have place the file sendDataToPostgres.py in flow folder where flow.py is also present in flow.py i am importing the function sendDataToOfflineStore but when i do quickrun the it fails and in logs i get the following error ModuleNotFoundError("No module named 'sendDataToPostgres'")

@Aqib Fayyaz you need to either place this function inside of your flow.py (importing it is not enough), or you need to include this module in your Docker image that you pass to your KubernetesRun.

ok Great i am going to look into this.

Yes, you create a Dockerfile, then you build your Docker image and push it to the container registry. And then, you can use it in your KubernetesRun provided that your Kubernetes cluster is authenticated to pull the image from that registry (e.g. using Kubernetes Secret). Which registry do you use - was it GCR?

yes i am going to use gcr

This might be helpful: https://blog.container-solutions.com/using-google-container-registry-with-kubernetes

Great Thank you so much.

@Aqib Fayyaz no, you don’t need to. Kubernetes agent is running in one pod, and the Secret is just another Kubernetes object within the cluster. Nice work!

Thank you Anna. but i have another issue now how to specify the image for gcr in image tag in kubernets run?

so far I have examples of AWS ECR and Azure ACR, bot not yet GCP’s Container Registry 😅 I will share as soon as I manage to build some example. But it should work pretty much the same way as for other cloud providers: you provide as image a string in the format

{REGISTRY_URL}/{image_name}

and the image_name may optionally contain a tag e.g.

image_name:latest

run_config=KubernetesRun(
        image="<http://prefectdemos.azurecr.io/community/flows|prefectdemos.azurecr.io/community/flows>",
        labels=["aks"],
        image_pull_secrets=["aks"],  # see README
    ),

AWS:

run_config=KubernetesRun(
        image=f"{AWS_ACCOUNT_ID}.<http://dkr.ecr.eu-central-1.amazonaws.com/community:latest|dkr.ecr.eu-central-1.amazonaws.com/community:latest>",
        labels=["k8s"],
        image_pull_secrets=["aws-ecr-secret"],  # see README
    ),

in image pull secret i have to give the kubernets secret name?

exactly

Pod prefect-job-80270896-hxw84 failed. Container 'flow' state: terminated Exit Code:: 1 Reason: Error

interesting. Can you share your run configuration?

run_config=KubernetesRun(image=f"gcr.io/superset-p4/xaimpl-anylyzer-prefectfeast:latest",image_pull_secrets=["gcr-json-key"]

it actually looks good. I will try reproduce and get back to you once I have a working example.

how can i know that issue is not related to kubernetes secret

great question. i think that the error would be something like:

Message: Failed to pull image "<http://XXXXXXX.dkr.ecr.eu-central-1.amazonaws.com/community:latest|XXXXXXX.dkr.ecr.eu-central-1.amazonaws.com/community:latest>": rpc error: code = Unknown desc = Error response from daemon: pull access denied for XXXX

when i used command kubectl logs PON_NAME i see following error File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.64.16.1', port=443): Max retries exceeded with url: /apis/batch/v1/namespaces/default/jobs?continue=&labelSelector=prefect.io%2Fidentifier&limit=20 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f329d2eef28>: Failed to establish a new connection: [Errno 111] Connection refused',)) ERRORagentError while managing existing k8s jobs

either the K8s job cannot pull the image from the host 10.64.16.1 or your agent doesn’t have permission to create a K8 job… I will look more. Please share if you find anything that may help in the meantime

ok

I couldn’t reproduce the issue when using GCR image with a local Kubernetes cluster, so I think it may either be: • permissions on your Kubernetes Role of the agent • your service account configuration for GCR I would check those two. Here is a working example for GCR configuration with Kubernetes secret setup https://github.com/anna-geller/packaging-prefect-flows/#google-container-registry and usage example in a flow: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run_custom_gcr_image.py

Great thank you for the Great help and efforts i'll look into this.

So if you want to use Docker agent instead of Kubernetes agent, and you use a GCR image (rather than a local one) in your flow, then you need to “docker login” to authenticate the terminal session of the docker agent so that the agent can pull images. This is a security mechanism to ensure that not everybody in the world can pull your private container images, but only you, if you authenticate with valid registry credentials.

hi, how can i check my default service account in terminal. Maybe i am using keys from different service account and my default service account is not that one

@Aqib Fayyaz I think you can activate it in your terminal by passing the path to your JSON key:

gcloud auth activate-service-account --key-file=your_file.json

while looking into logs i also came accros this error File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 323, in _submit_deploy_flow_run_jobs flow_run_ids = self._get_ready_flow_runs() File "/usr/local/lib/python3.6/site-packages/prefect/agent/agent.py", line 580, in _get_ready_flow_runs "tenant_id": self.client.tenant_id, File "/usr/local/lib/python3.6/site-packages/prefect/client/client.py", line 561, in graphql raise AuthorizationError(result["errors"]) prefect.exceptions.AuthorizationError: [{'path': ['get_runs_in_queue'], 'message': 'AuthenticationError: Forbidden', 'extensions': {'code': 'UNAUTHENTICATED'}}]

Did you add an API key to your Kubernetes agent? the error looks like the API key is invalid or expired

i did

so the Prefect agent is deployed as one pod in a Kubernetes cluster. Your GCR Kubernetes Secret is another Kubernetes object, completely independent of the agent pod. So adding Secrets doesn’t require you to redeploy the agent

Thank you @Anna Geller for all the Great help.