Lone Pine Account
05/14/2021, 5:14 PMLone Pine Account
05/14/2021, 5:14 PMfrom prefect import task, Flow
from prefect.tasks.secrets import PrefectSecret
from prefect.tasks.databricks import DatabricksRunNow
import requests
conncfg = {"host":"<https://xxx.cloud.databricks.com/>", "token":"xxx"}
job_json = {
"job_id": 24242,
"notebook_params": {}
}
@task
def run_databricks(job_config):
notebook_run = DatabricksRunNow(json=job_config).run()
run_id = notebook_run(databricks_conn_secret=conncfg)
headers = { "Authorization" : "Bearer xxx" }
r = requests.get('<https://xxx.cloud.databricks.com/api/2.0/jobs/runs/get-output?run_id=%s'%(runid)>, headers=headers)
results = r.json()
output = results['metadata']['notebook_output']
if output.has_key('message'):
raise Exception(output['message'])
with Flow('testflow') as flow:
run_databricks(job_json)
Lone Pine Account
05/14/2021, 5:15 PMLone Pine Account
05/14/2021, 5:15 PMUnexpected error: AssertionError('A databricks connection string must be supplied as a dictionary or through Prefect Secrets')
Lone Pine Account
05/14/2021, 5:16 PMRobert Bastian
05/14/2021, 8:48 PMhaf
05/15/2021, 2:01 PMSecrets page
in the cloud UI: I know I've been there, but I can't for the life of me find my way back there... Where is it? Could someone provide a link? Google says nothing and the docs link back to itself.haf
05/15/2021, 2:16 PMhaf
05/15/2021, 2:38 PMhaf
05/15/2021, 2:43 PMKha Nguyen
05/15/2021, 4:42 PMexecutor = DaskExecutor(
cluster_class="dask_cloudprovider.FargateCluster",
cluster_kwargs={
"image": "prefecthq/prefect:latest",
"n_workers": 5,
...
},
)
Does it mean that Prefect can also create a Dask cluster when there is some flows to run, and tear down the Dask cluster on Fargate once all flows have been run? I already have a setup using ECSAgent. Does this executor mean that I don’t need that ECS agent any longer?Kha Nguyen
05/15/2021, 4:44 PMMatej
05/16/2021, 8:10 AMfrom prefect import task
@task
def add(x, y):
return x + y
add(1, 2) # 3
this example throws me:
ValueError: Could not infer an active Flow context while creating edge
and suggests using add.run() instead. In the tutorial however, they use add(1,2) regularly.
Are the tutorials outdated for the newest version of prefect?
edit: I am also unable to use .map as it is used in the tutorial:
add.map([1, 10], [2, 3])
throws the same error? How am I supposed to experiment with tasks without flow context?Matej
05/16/2021, 8:50 AMMark McDonald
05/16/2021, 1:26 PMPedro Machado
05/17/2021, 2:42 AMStéphan Taljaard
05/17/2021, 7:09 AMElliot Oram
05/17/2021, 9:03 AM...
/home/rof/.pyenv/versions/3.9.1/lib/python3.9/site-packages/marshmallow/fields.py:198
/home/rof/.pyenv/versions/3.9.1/lib/python3.9/site-packages/marshmallow/fields.py:198: RemovedInMarshmallow4Warning: Passing field metadata as a keyword arg is deprecated. Use the explicit `metadata=...` argument instead.
...
Tried to update to the latest version of prefect and still no joy!Domantas
05/17/2021, 12:06 PMjob_template_path
where there is a path to yaml file which is supposed to create environment variables and used them in the prefect code:
apiVersion: v1
kind: Pod
metadata:
name: secret-test-pod
spec:
containers:
- name: test-container
image: <http://k8s.gcr.io/busybox|k8s.gcr.io/busybox>
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- secretRef:
name: mysecret
restartPolicy: Never
Yaml file template is took from kubernetes documentation(https://kubernetes.io/docs/concepts/configuration/secret/#use-cases)
However, when I try to run it on prefect server, I got this error: "'spec'"
(will attach an error log screenshot)
Is there are anything wrong with the template?
Is this approach is a good way to define and use secrets in the kubernetes agent?
P.S I tried to define env
parameter with secrets by using KubernetesRun agent but it is not a good option for this case since all secrets could be visible as a plain text in the prefect server UI.Pedro Machado
05/17/2021, 1:29 PMZach Schumacher
05/17/2021, 3:09 PMjob_template
in my KubernetesRun
config, I’m stilling seeing default set in the UI. I believe previously I would see the template there. Anyone know what is going on? We are running prefect cloud and my core version is 0.14.19
Trevor Kramer
05/17/2021, 3:35 PMJoseph Loss
05/17/2021, 3:56 PMhaf
05/17/2021, 4:13 PMDamien Ramunno-Johnson
05/17/2021, 4:52 PMYour team has no users available; upgrade your plan to get more users and access to more features!
Any ideas?DK
05/17/2021, 6:38 PMEnda Peng
05/17/2021, 8:20 PMflow.register()
function? I notice that if I use prefect register
command line tool, the label is exactly what I provided, but if I use python code, it will be added. I read the doc and it says this is the desired behavior, wonder if there is a way to get it removed?Damien Ramunno-Johnson
05/17/2021, 10:17 PMA result subclass that matches the storage backend of yourDoes this mean we can not use checkpointing if using Docker Storage?setting will automatically be applied to all tasks, if available; notably this is not yet supported for Docker Storageprefect.config.flows.storage
Walee
05/17/2021, 11:22 PMTypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
[2021-05-17 23:14:44,507] ERROR - agent | Error while managing existing k8s jobs
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 383, in heartbeat
self.manage_jobs()
File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 236, in manage_jobs
pod_name
TypeError: '<' not supported between instances of 'NoneType' and 'datetime.datetime'
we’re using the config generated from the prefect cli for kubernetes agent. Any idea on how to work resolve it?Joseph Loss
05/18/2021, 2:14 AMJoseph Loss
05/18/2021, 2:14 AM