Sean Talia
01/28/2021, 5:30 PMtask_definition_arn
, task_definition_path
, task_definition
) to the flow considered to be canonical? i'm trying to weigh the pros and cons of having some other process like terraform handle management of the task definition (thereby only leaving the ARN to be supplied to prefect) vs. using prefect to handle thatJosh Greenhalgh
01/28/2021, 7:09 PMraaid
01/28/2021, 9:39 PMFelipe Saldana
01/28/2021, 9:57 PMUnexpected error: TypeError("Object of type 'TestTask' is not JSON serializable",)
TestTask is a custom task type I created and the run method returns a custom return type I created as well.Adam Brusselback
01/28/2021, 11:43 PMJan Marais
01/29/2021, 6:05 AMDJ Erraballi
01/29/2021, 6:42 AMTim Pörtner
01/29/2021, 9:33 AMBorut Hafner
01/29/2021, 1:29 PMAlex Furrier
01/29/2021, 3:57 PM@task
def get_array_input(db_client, query):
return db_client.query(query)
@task
def compute_array_write_to_db(input):
array = requests.get_huge_array()
db_client.write(array)
with Flow("generate-and-write-arrays",) as generate_and_write_arrays:
lots_of_inputs = get_array_input(db_client, query)
lots_of_inputs.map(compute_array_write_to_db)
# Crashes after certain amount of arrays
# due to memory limit
generate_and_write_arrays.run()
jeff n
01/29/2021, 4:33 PMwith Flow("Example") as flow:
....
def __main__():
if os.getenv("production") == True:
flow.schedule(my_scheduler)
flow.register()
If I register a flow with that in the main would I be able to control if it has a schedule?Levi Leal
01/29/2021, 4:45 PMJosh
01/29/2021, 5:06 PMFailed to write log with error: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe')))
In Prefect it says the task failed, but it still seems to be executing in the agent. There were logs before from the agent to the cloud, but no logs afterwardsKieran
01/29/2021, 5:24 PMPrefect automatically gathers mapped results into a list if they are needed by a non-mapped task
I have two mapped tasks, one feeding the other, then a third non-mapped task which expects a list. The logger context doesn't appear to be getting passed so I cant get any insight. The UI is showing the child-mapped tasks and I have logged out their content but their results are not being gathered together as suggested in the docs.
Any pointers to help would be amazing!Raphaël Riel
01/29/2021, 7:25 PM<http://univeral.prefet.io|univeral.prefet.io>
D) Click on Google Again.
E) Now I have Google’s prompt to select which account to use (Personal, Work, etc.)
F) Then I’m logged in.
Any idea? Can this be related to https://prefect-community.slack.com/archives/CL09KU1K7/p1611786977326500 ?Josh
01/30/2021, 3:32 AMFailed to write log with error: 413 Client Error: Request Entity Too Large for url: <https://api.prefect.io/graphql>
Peter Roelants
01/30/2021, 8:36 AM"test-volume"
created with docker volume create test-volume
If I want to mount the named docker volume "test-volume"
with prefect agent docker start --volume test-volume:$DOCKER_DIR ...
, the named volume is never mounted into the runtime started by the agent, a new volume seems to be created and mounted each time.jspeis
01/31/2021, 1:43 AMRyan Kelly
01/31/2021, 3:29 PMAnatoliy Zhyzhkevych
01/31/2021, 9:51 PMPeyton Runyan
01/31/2021, 10:58 PMmode
as a parameter instead of messing with it in the config and re-registering my flow.
mode = "hard_coded_mode"
[sql_server]
server = server
driver = driver
dsn = "MYMSSQL"
user = "${sql_server.${mode}.user}"
database = "${sql_server.${mode}.database}"
[sql_server.dev]
user = dev user
database = "dev db
[sql_server.hub]
user = app user
database = app db
[sql_server.prod]
user = prod user
database = prod db
Yash Bhandari
02/01/2021, 1:34 AMMichael Hadorn
02/01/2021, 8:25 AMAdam
02/01/2021, 12:15 PMGiovanni Giacco
02/01/2021, 1:06 PMAdrien Boutreau
02/01/2021, 3:17 PMJoël Luijmes
02/01/2021, 3:27 PM@pytest.fixture
def mock_resource_manager(monkeypatch):
mock = MagicMock(return_value=None)
monkeypatch.setattr("prefect.tasks.core.resource_manager.ResourceManager.__call__", mock)
return mock
But this gives weird errors down the road, as I also want to mock prefect.config/context for other purposes.Josh Greenhalgh
02/01/2021, 4:16 PMKilian
02/01/2021, 4:43 PMMorgan Omind
02/01/2021, 4:58 PMMorgan Omind
02/01/2021, 4:58 PM{
"insertId": "fl6tfmd7rpcqwqaig",
"jsonPayload": {
"_SYSTEMD_UNIT": "kubelet.service",
"_SYSTEMD_SLICE": "system.slice",
"_CAP_EFFECTIVE": "3fffffffff",
"_HOSTNAME": "gke-iguazu-default-pool-f66089ee-s1lr",
"SYSLOG_FACILITY": "3",
"_PID": "1087",
"_SYSTEMD_INVOCATION_ID": "3eebf72f9a3b413681b38725870a2629",
"_BOOT_ID": "fa0576989dd2496b8c759f2a2e34514a",
"_GID": "0",
"_STREAM_ID": "d30fb7ed4a014871a2f8970efe80c5ad",
"SYSLOG_IDENTIFIER": "kubelet",
"PRIORITY": "6",
"_TRANSPORT": "stdout",
"_COMM": "kubelet",
"MESSAGE": "E0201 16:20:44.424380 1087 pod_workers.go:191] Error syncing pod 171b3eee-639d-4da7-9deb-a155f1c87c4a (\"prefect-job-a2802066-vjmf9_default(171b3eee-639d-4da7-9deb-a155f1c87c4a)\"), skipping: failed to \"StartContainer\" for \"flow\" with ImagePullBackOff: \"Back-off pulling image \\\"<http://eu.gcr.io/quetzal-omind/iguazu/dev/flows/bilan_vr:2020-09-17t18-14-03-371762-00-00|eu.gcr.io/quetzal-omind/iguazu/dev/flows/bilan_vr:2020-09-17t18-14-03-371762-00-00>\\\"\"",
"_MACHINE_ID": "6f4f58b7702e59ad315033846174af48",
"_SYSTEMD_CGROUP": "/system.slice/kubelet.service",
"_EXE": "/home/kubernetes/bin/kubelet",
"_UID": "0",
"_CMDLINE": "/home/kubernetes/bin/kubelet --v=2 --cloud-provider=gce --experimental-check-node-capabilities-before-mount=true --experimental-mounter-path=/home/kubernetes/containerized_mounter/mounter --cert-dir=/var/lib/kubelet/pki/ --cni-bin-dir=/home/kubernetes/bin --kubeconfig=/var/lib/kubelet/kubeconfig --image-pull-progress-deadline=5m --experimental-kernel-memcg-notification=true --max-pods=110 --non-masquerade-cidr=0.0.0.0/0 --network-plugin=kubenet --node-labels=<http://cloud.google.com/gke-nodepool=default-pool,cloud.google.com/gke-os-distribution=cos|cloud.google.com/gke-nodepool=default-pool,cloud.google.com/gke-os-distribution=cos> --volume-plugin-dir=/home/kubernetes/flexvolume --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --node-status-max-images=25 --registry-qps=10 --registry-burst=20 --config /home/kubernetes/kubelet-config.yaml --pod-sysctls=net.core.somaxconn=1024,net.ipv4.conf.all.accept_redirects=0,net.ipv4.conf.all.forwarding=1,net.ipv4.conf.all.route_localnet=1,net.ipv4.conf.default.forwarding=1,net.ipv4.ip_forward=1,net.ipv4.tcp_fin_timeout=60,net.ipv4.tcp_keepalive_intvl=75,net.ipv4.tcp_keepalive_probes=9,net.ipv4.tcp_keepalive_time=7200,net.ipv4.tcp_rmem=4096 87380 6291456,net.ipv4.tcp_syn_retries=6,net.ipv4.tcp_tw_reuse=0,net.ipv4.tcp_wmem=4096 16384 4194304,net.ipv4.udp_rmem_min=4096,net.ipv4.udp_wmem_min=4096,net.ipv6.conf.default.accept_ra=0,<http://net.netfilter.nf|net.netfilter.nf>_conntrack_generic_timeout=600,<http://net.netfilter.nf|net.netfilter.nf>_conntrack_tcp_timeout_close_wait=3600,<http://net.netfilter.nf|net.netfilter.nf>_conntrack_tcp_timeout_established=86400"
},
"resource": {
"type": "k8s_node",
"labels": {
"cluster_name": "iguazu",
"project_id": "cartographie-cps",
"node_name": "gke-iguazu-default-pool-f66089ee-s1lr",
"location": "europe-west4-a"
}
},
"timestamp": "2021-02-01T16:20:44.424424Z",
"labels": {
"<http://gke.googleapis.com/log_type|gke.googleapis.com/log_type>": "system"
},
"logName": "projects/cartographie-cps/logs/kubelet",
"receiveTimestamp": "2021-02-01T16:20:49.212486852Z"
}
Kyle Moon-Wright
02/01/2021, 8:54 PM"cartographie-cps"
, and the only reference to quetzal-omind
I see is your storage method - containerized code retrieved from your GCR registry. Your Agent will pull the image from this registry URL at runtime for execution inside it’s environment. Keep in mind that your code storage method is independent from your Prefect Project directory in your tenant.
I’m not sure how best to help here, but happy to answer any questions you may have to clarify further.Morgan Omind
02/01/2021, 9:09 PMKyle Moon-Wright
02/01/2021, 9:20 PMpip install prefect[kubernetes]
and maybe
pip install prefect[google]
as optional dependencies as described here.
Otherwise, your Agent doesn’t need access to a specified Prefect Project (which has very little bearing on execution) but it does need access to the GCP registry to pull the image, as well as other RBAC /network permissions on the GCP side to submit Jobs as containers from the image you pulled.Morgan Omind
02/02/2021, 9:51 AM15:47:20
DEBUG
urllib3.connectionpool
Starting new HTTPS connection (1): <http://api.prefect.io:443|api.prefect.io:443>
15:47:20
DEBUG
urllib3.connectionpool
<https://api.prefect.io:443> "POST /graphql HTTP/1.1" 200 45
{
"textPayload": "[2021-02-02 10:44:58,462] DEBUG - agent | No flow runs found\n",
"insertId": "28isj4g32dpqfj",
"resource": {
"type": "k8s_container",
"labels": {
"namespace_name": "default",
"container_name": "k8s-agent",
"pod_name": "prefect-1612261772-k8s-agent-76968b94dc-vrb48",
"project_id": "cartographie-cps",
"cluster_name": "iguazu",
"location": "europe-west1-c"
}
},
"timestamp": "2021-02-02T10:44:58.463191937Z",
"severity": "INFO",
"labels": {
"<http://compute.googleapis.com/resource_name|compute.googleapis.com/resource_name>": "gke-iguazu-default-pool-4df57660-prlp",
"k8s-pod/app": "prefect",
"k8s-pod/app_kubernetes_io/version": "0.10.7",
"k8s-pod/app_kubernetes_io/instance": "prefect-1612261772",
"k8s-pod/pod-template-hash": "76968b94dc",
"k8s-pod/app_kubernetes_io/component": "k8s-agent",
"k8s-pod/app_kubernetes_io/managed-by": "Helm",
"k8s-pod/helm_sh/chart": "prefect-0.1.0",
"k8s-pod/app_kubernetes_io/name": "prefect"
},
"logName": "projects/cartographie-cps/logs/stdout",
"receiveTimestamp": "2021-02-02T10:45:02.024152388Z"
}
{
"textPayload": "[2021-02-02 10:48:19,205] DEBUG - agent | Querying for flow runs\n",
"insertId": "jzrqvyg3j63ez5",
"resource": {
"type": "k8s_container",
"labels": {
"project_id": "cartographie-cps",
"container_name": "k8s-agent",
"namespace_name": "default",
"location": "europe-west1-c",
"cluster_name": "iguazu",
"pod_name": "prefect-1612261772-k8s-agent-76968b94dc-vrb48"
}
},
"timestamp": "2021-02-02T10:48:19.208557067Z",
"severity": "INFO",
"labels": {
"k8s-pod/helm_sh/chart": "prefect-0.1.0",
"k8s-pod/pod-template-hash": "76968b94dc",
"k8s-pod/app_kubernetes_io/component": "k8s-agent",
"k8s-pod/app": "prefect",
"<http://compute.googleapis.com/resource_name|compute.googleapis.com/resource_name>": "gke-iguazu-default-pool-4df57660-prlp",
"k8s-pod/app_kubernetes_io/name": "prefect",
"k8s-pod/app_kubernetes_io/instance": "prefect-1612261772",
"k8s-pod/app_kubernetes_io/managed-by": "Helm",
"k8s-pod/app_kubernetes_io/version": "0.10.7"
},
"logName": "projects/cartographie-cps/logs/stdout",
"receiveTimestamp": "2021-02-02T10:48:22.022280350Z"
}
Florent VanDeMoortele
02/02/2021, 5:35 PMKyle Moon-Wright
02/02/2021, 6:50 PMFlorent VanDeMoortele
02/03/2021, 9:56 AM