Lone Pine Account
05/14/2021, 5:16 PMRobert Bastian
05/14/2021, 8:48 PMhaf
05/15/2021, 2:01 PMSecrets page
in the cloud UI: I know I've been there, but I can't for the life of me find my way back there... Where is it? Could someone provide a link? Google says nothing and the docs link back to itself.haf
05/15/2021, 2:16 PMhaf
05/15/2021, 2:38 PMhaf
05/15/2021, 2:43 PMKha Nguyen
05/15/2021, 4:42 PMexecutor = DaskExecutor(
cluster_class="dask_cloudprovider.FargateCluster",
cluster_kwargs={
"image": "prefecthq/prefect:latest",
"n_workers": 5,
...
},
)
Does it mean that Prefect can also create a Dask cluster when there is some flows to run, and tear down the Dask cluster on Fargate once all flows have been run? I already have a setup using ECSAgent. Does this executor mean that I don’t need that ECS agent any longer?Kha Nguyen
05/15/2021, 4:44 PMMatej
05/16/2021, 8:10 AMfrom prefect import task
@task
def add(x, y):
return x + y
add(1, 2) # 3
this example throws me:
ValueError: Could not infer an active Flow context while creating edge
and suggests using add.run() instead. In the tutorial however, they use add(1,2) regularly.
Are the tutorials outdated for the newest version of prefect?
edit: I am also unable to use .map as it is used in the tutorial:
add.map([1, 10], [2, 3])
throws the same error? How am I supposed to experiment with tasks without flow context?Matej
05/16/2021, 8:50 AMMark McDonald
05/16/2021, 1:26 PMPedro Machado
05/17/2021, 2:42 AMStéphan Taljaard
05/17/2021, 7:09 AMElliot Oram
05/17/2021, 9:03 AM...
/home/rof/.pyenv/versions/3.9.1/lib/python3.9/site-packages/marshmallow/fields.py:198
/home/rof/.pyenv/versions/3.9.1/lib/python3.9/site-packages/marshmallow/fields.py:198: RemovedInMarshmallow4Warning: Passing field metadata as a keyword arg is deprecated. Use the explicit `metadata=...` argument instead.
...
Tried to update to the latest version of prefect and still no joy!Domantas
05/17/2021, 12:06 PMjob_template_path
where there is a path to yaml file which is supposed to create environment variables and used them in the prefect code:
apiVersion: v1
kind: Pod
metadata:
name: secret-test-pod
spec:
containers:
- name: test-container
image: <http://k8s.gcr.io/busybox|k8s.gcr.io/busybox>
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- secretRef:
name: mysecret
restartPolicy: Never
Yaml file template is took from kubernetes documentation(https://kubernetes.io/docs/concepts/configuration/secret/#use-cases)
However, when I try to run it on prefect server, I got this error: "'spec'"
(will attach an error log screenshot)
Is there are anything wrong with the template?
Is this approach is a good way to define and use secrets in the kubernetes agent?
P.S I tried to define env
parameter with secrets by using KubernetesRun agent but it is not a good option for this case since all secrets could be visible as a plain text in the prefect server UI.Pedro Machado
05/17/2021, 1:29 PMZach Schumacher
05/17/2021, 3:09 PMjob_template
in my KubernetesRun
config, I’m stilling seeing default set in the UI. I believe previously I would see the template there. Anyone know what is going on? We are running prefect cloud and my core version is 0.14.19
Trevor Kramer
05/17/2021, 3:35 PMJoseph Loss
05/17/2021, 3:56 PMhaf
05/17/2021, 4:13 PMDamien Ramunno-Johnson
05/17/2021, 4:52 PMYour team has no users available; upgrade your plan to get more users and access to more features!
Any ideas?DK
05/17/2021, 6:38 PMEnda Peng
05/17/2021, 8:20 PMflow.register()
function? I notice that if I use prefect register
command line tool, the label is exactly what I provided, but if I use python code, it will be added. I read the doc and it says this is the desired behavior, wonder if there is a way to get it removed?Damien Ramunno-Johnson
05/17/2021, 10:17 PMA result subclass that matches the storage backend of yourDoes this mean we can not use checkpointing if using Docker Storage?setting will automatically be applied to all tasks, if available; notably this is not yet supported for Docker Storageprefect.config.flows.storage
Walee
05/17/2021, 11:22 PMTypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
[2021-05-17 23:14:44,507] ERROR - agent | Error while managing existing k8s jobs
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 383, in heartbeat
self.manage_jobs()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 236, in manage_jobs
pod_name
TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
we’re using the config generated from the prefect cli for kubernetes agent. Any idea on how to work resolve it?Joseph Loss
05/18/2021, 2:14 AMVaibhav Shetye
05/18/2021, 3:33 AM[Errno 11] Resource temporarily unavailable
any hints on what could be the underlying issue?Jacob Blanco
05/18/2021, 5:58 AM"my_table-intrepid-iguana"
or something like that.
This would make it easier to navigate the flow runs in the Cloud UI.Chris McClellan
05/18/2021, 7:34 AMDomantas
05/18/2021, 12:32 PMjob_template_path
parameter in KubernetesRun
by passing yaml file which looks like this:
apiVersion: batch/v1
kind: Job
metadata:
name: prefect-job
labels: {}
spec:
template:
metadata:
labels: {}
spec:
containers:
- name: flow
image: test-image:1.2
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c"]
args: ["prefect execute flow-run"]
env:
- name: PREFECT__CONTEXT__SECRETS__database_psw
value: veryStrongPasswrod
volumeMounts:
- name: storage
mountPath: /tnt/test/domantas_test
restartPolicy: Never
imagePullSecrets:
- name: registry-secret
volumes:
- name: storage
cephfs:
monitors:
- 59.11.129.131
- 59.11.129.132
- 59.11.129.133
- 59.11.129.134
- 59.11.129.135
path: /test
user: storage
secretRef:
name: ceph-secret
However, when I try to execute in prefect server UI, it seems the flow get stuck with "Submitted for execution" status with no useful logs.
I took job_template example from here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/kubernetes/job_spec.yaml and additionally added cephFS necessary parameters.