Morgan Omind
02/01/2021, 4:58 PM{
"insertId": "fl6tfmd7rpcqwqaig",
"jsonPayload": {
"_SYSTEMD_UNIT": "kubelet.service",
"_SYSTEMD_SLICE": "system.slice",
"_CAP_EFFECTIVE": "3fffffffff",
"_HOSTNAME": "gke-iguazu-default-pool-f66089ee-s1lr",
"SYSLOG_FACILITY": "3",
"_PID": "1087",
"_SYSTEMD_INVOCATION_ID": "3eebf72f9a3b413681b38725870a2629",
"_BOOT_ID": "fa0576989dd2496b8c759f2a2e34514a",
"_GID": "0",
"_STREAM_ID": "d30fb7ed4a014871a2f8970efe80c5ad",
"SYSLOG_IDENTIFIER": "kubelet",
"PRIORITY": "6",
"_TRANSPORT": "stdout",
"_COMM": "kubelet",
"MESSAGE": "E0201 16:20:44.424380 1087 pod_workers.go:191] Error syncing pod 171b3eee-639d-4da7-9deb-a155f1c87c4a (\"prefect-job-a2802066-vjmf9_default(171b3eee-639d-4da7-9deb-a155f1c87c4a)\"), skipping: failed to \"StartContainer\" for \"flow\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://eu.gcr.io/quetzal-omind/iguazu/dev/flows/bilan_vr:2020-09-17t18-14-03-371762-00-00|eu.gcr.io/quetzal-omind/iguazu/dev/flows/bilan_vr:2020-09-17t18-14-03-371762-00-00>\\\"\"",
"_MACHINE_ID": "6f4f58b7702e59ad315033846174af48",
"_SYSTEMD_CGROUP": "/system.slice/kubelet.service",
"_EXE": "/home/kubernetes/bin/kubelet",
"_UID": "0",
"_CMDLINE": "/home/kubernetes/bin/kubelet --v=2 --cloud-provider=gce --experimental-check-node-capabilities-before-mount=true --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --cert-dir=/var/lib/kubelet/pki/ --cni-bin-dir=/home/kubernetes/bin --kubeconfig=/var/lib/kubelet/kubeconfig --image-pull-progress-deadline=5m --experimental-kernel-memcg-notification=true --max-pods=110 --non-masquerade-cidr=0.0.0.0/0 --network-plugin=kubenet --node-labels=<http://cloud.google.com/gke-nodepool=default-pool,cloud.google.com/gke-os-distribution=cos|cloud.google.com/gke-nodepool=default-pool,cloud.google.com/gke-os-distribution=cos> --volume-plugin-dir=/home/kubernetes/flexvolume --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --node-status-max-images=25 --registry-qps=10 --registry-burst=20 --config /home/kubernetes/kubelet-config.yaml --pod-sysctls=net.core.somaxconn=1024,net.ipv4.conf.all.accept_redirects=0,net.ipv4.conf.all.forwarding=1,net.ipv4.conf.all.route_localnet=1,net.ipv4.conf.default.forwarding=1,net.ipv4.ip_forward=1,net.ipv4.tcp_fin_timeout=60,net.ipv4.tcp_keepalive_intvl=75,net.ipv4.tcp_keepalive_probes=9,net.ipv4.tcp_keepalive_time=7200,net.ipv4.tcp_rmem=4096 87380 6291456,net.ipv4.tcp_syn_retries=6,net.ipv4.tcp_tw_reuse=0,net.ipv4.tcp_wmem=4096 16384 4194304,net.ipv4.udp_rmem_min=4096,net.ipv4.udp_wmem_min=4096,net.ipv6.conf.default.accept_ra=0,<http://net.netfilter.nf|net.netfilter.nf>_conntrack_generic_timeout=600,<http://net.netfilter.nf|net.netfilter.nf>_conntrack_tcp_timeout_close_wait=3600,<http://net.netfilter.nf|net.netfilter.nf>_conntrack_tcp_timeout_established=86400"
},
"resource": {
"type": "k8s_node",
"labels": {
"cluster_name": "iguazu",
"project_id": "cartographie-cps",
"node_name": "gke-iguazu-default-pool-f66089ee-s1lr",
"location": "europe-west4-a"
}
},
"timestamp": "2021-02-01T16:20:44.424424Z",
"labels": {
"<http://gke.googleapis.com/log_type|gke.googleapis.com/log_type>": "system"
},
"logName": "projects/cartographie-cps/logs/kubelet",
"receiveTimestamp": "2021-02-01T16:20:49.212486852Z"
}
Kyle Moon-Wright
02/01/2021, 8:54 PM"cartographie-cps"
, and the only reference to quetzal-omind
I see is your storage method - containerized code retrieved from your GCR registry. Your Agent will pull the image from this registry URL at runtime for execution inside it’s environment. Keep in mind that your code storage method is independent from your Prefect Project directory in your tenant.
I’m not sure how best to help here, but happy to answer any questions you may have to clarify further.Morgan Omind
02/01/2021, 9:09 PMKyle Moon-Wright
02/01/2021, 9:20 PMpip install prefect[kubernetes]
and maybe
pip install prefect[google]
as optional dependencies as described here.
Otherwise, your Agent doesn’t need access to a specified Prefect Project (which has very little bearing on execution) but it does need access to the GCP registry to pull the image, as well as other RBAC /network permissions on the GCP side to submit Jobs as containers from the image you pulled.Morgan Omind
02/02/2021, 9:51 AM15:47:20
DEBUG
urllib3.connectionpool
Starting new HTTPS connection (1): <http://api.prefect.io:443|api.prefect.io:443>
15:47:20
DEBUG
urllib3.connectionpool
<https://api.prefect.io:443> "POST /graphql HTTP/1.1" 200 45
{
"textPayload": "[2021-02-02 10:44:58,462] DEBUG - agent | No flow runs found\n",
"insertId": "28isj4g32dpqfj",
"resource": {
"type": "k8s_container",
"labels": {
"namespace_name": "default",
"container_name": "k8s-agent",
"pod_name": "prefect-1612261772-k8s-agent-76968b94dc-vrb48",
"project_id": "cartographie-cps",
"cluster_name": "iguazu",
"location": "europe-west1-c"
}
},
"timestamp": "2021-02-02T10:44:58.463191937Z",
"severity": "INFO",
"labels": {
"<http://compute.googleapis.com/resource_name|compute.googleapis.com/resource_name>": "gke-iguazu-default-pool-4df57660-prlp",
"k8s-pod/app": "prefect",
"k8s-pod/app_kubernetes_io/version": "0.10.7",
"k8s-pod/app_kubernetes_io/instance": "prefect-1612261772",
"k8s-pod/pod-template-hash": "76968b94dc",
"k8s-pod/app_kubernetes_io/component": "k8s-agent",
"k8s-pod/app_kubernetes_io/managed-by": "Helm",
"k8s-pod/helm_sh/chart": "prefect-0.1.0",
"k8s-pod/app_kubernetes_io/name": "prefect"
},
"logName": "projects/cartographie-cps/logs/stdout",
"receiveTimestamp": "2021-02-02T10:45:02.024152388Z"
}
{
"textPayload": "[2021-02-02 10:48:19,205] DEBUG - agent | Querying for flow runs\n",
"insertId": "jzrqvyg3j63ez5",
"resource": {
"type": "k8s_container",
"labels": {
"project_id": "cartographie-cps",
"container_name": "k8s-agent",
"namespace_name": "default",
"location": "europe-west1-c",
"cluster_name": "iguazu",
"pod_name": "prefect-1612261772-k8s-agent-76968b94dc-vrb48"
}
},
"timestamp": "2021-02-02T10:48:19.208557067Z",
"severity": "INFO",
"labels": {
"k8s-pod/helm_sh/chart": "prefect-0.1.0",
"k8s-pod/pod-template-hash": "76968b94dc",
"k8s-pod/app_kubernetes_io/component": "k8s-agent",
"k8s-pod/app": "prefect",
"<http://compute.googleapis.com/resource_name|compute.googleapis.com/resource_name>": "gke-iguazu-default-pool-4df57660-prlp",
"k8s-pod/app_kubernetes_io/name": "prefect",
"k8s-pod/app_kubernetes_io/instance": "prefect-1612261772",
"k8s-pod/app_kubernetes_io/managed-by": "Helm",
"k8s-pod/app_kubernetes_io/version": "0.10.7"
},
"logName": "projects/cartographie-cps/logs/stdout",
"receiveTimestamp": "2021-02-02T10:48:22.022280350Z"
}
Florent VanDeMoortele
02/02/2021, 5:35 PMKyle Moon-Wright
02/02/2021, 6:50 PMFlorent VanDeMoortele
02/03/2021, 9:56 AM