microft
08/16/2022, 2:44 PMTim Enders
08/16/2022, 4:40 PMNone
defined on a mapped task? I now get this error (TypeError: object of type 'NoneType' has no len()
) when trying to map across the subsequent function signature.
@task
def get_items_list(
client, entity, page,
response_attr=None, path=None, params=None
):
Sam Garvis
08/16/2022, 5:45 PMSecret.load("slackbot-oauth").get()
, I get AttributeError: 'coroutine' object has no attribute 'get'
whether I run this in an ipynb or with a python file through the terminal.
Is it expected to run this command with await and async?
Or is this not supposed to happen?Blake Hamm
08/16/2022, 6:16 PMAlexander Belikov
08/16/2022, 6:19 PMmultiprocessing.Process
as
processes = []
for chunk in url_chunks:
kwargs["mp3_urls"] = chunk
processes.append(mp.Process(foo=foo, kwargs=kwargs))
for p in processes:
p.start()
for p in processes:
p.join()
and it worked. The main reason for using multiprocessing.Process
is to have a large model in memory shared between processes.
In fact it works when I run this exact flow with prefect via a python script calling the flow as a fucntion, it works as expected. However, when run it as prefect deployment run ...
or via a scheduled run, the processes in p.start()
don't run. The flow result is marked as success to my surprise.
Any ideas?chicago-joe
08/16/2022, 7:24 PMVlad Tudor
08/16/2022, 7:42 PMKubernetesAgent
locally and to bind it to minikube for testing purposes. For now, I see the Agent in the Server UI, but the tasks are not executed. How do I configure this? Thank you for you patience, I am new to Prefect.
๐ minikube v1.26.1 on Ubuntu 20.04
โจ Using the docker driver based on existing profile
๐ Starting control plane node minikube in cluster minikube
๐ Pulling base image ...
๐ Updating the running docker "minikube" container ...
๐ณ Preparing Kubernetes v1.24.3 on Docker 20.10.17 ...
โช Generating certificates and keys ...
โช Booting up control plane ...
โช Configuring RBAC rules ...
๐ Verifying Kubernetes components...
โช Using image <http://gcr.io/k8s-minikube/storage-provisioner:v5|gcr.io/k8s-minikube/storage-provisioner:v5>
๐ Enabled addons: storage-provisioner, default-storageclass
๐ Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Tim Enders
08/16/2022, 8:54 PMsqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: <https://sqlalche.me/e/14/3o7r>)
Hammad Haqqani
08/16/2022, 10:03 PMprefect agent ecs start --cluster arn:aws:ecs:us-west-2:00000000:cluster/prefect-dev --label aws-ecs
Usage: prefect agent [OPTIONS] COMMAND [ARGS]...
Try 'prefect agent --help' for help.
โญโ Error โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ No such command 'ecs'. โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Ben Muller
08/16/2022, 10:38 PMMars
08/16/2022, 11:57 PMif __name__ == '__main__': myflow()
. This is without a running Orion server.
The code in my flow that creates the Secret block appears to work. However, when I try use my Secret block in my 2.0 flow the prefect client raises a PrefectHTTPStatusError, and says โBlock document not foundโ.
In Prefect 1.0 I could use PREFECT__CONTEXT__SECRETS__
to set the values I need, but I donโt see any mention of similar functionality for 2.0.Jaime Raldua Veuthey
08/17/2022, 7:52 AMTim Helfensdรถrfer
08/17/2022, 9:37 AMTim Helfensdรถrfer
08/17/2022, 10:15 AMMohamed Ayoub Chettouh
08/17/2022, 10:29 AMprefect block type ls
does give the block types.
UPDATE2: Something is off, when hosting on localhost and doing an ssh tunnel on it it works like a charm. When hosting on 0.0.0.0
the page is empty.
UPDATE3: Now it works on the public IP as well ?!? Maybe it needs to start without --host
the first timeAndreas Nigg
08/17/2022, 11:27 AMprefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml
However, as soon as I add infrastr. overrides, the flow stays in "pending" state forever
prefect deployment build r_script_automation.py:r_script_automation --name retention_cohort_analysis_deployment -t k8s -sb gcs/gcs-prefect-stprage -ib kubernetes-job/generic-k8s-job -o cohort_deployment.yaml --override image=europe-docker.pkg.dev/vol-at/rm-datateam-repository/r-script-automation:beta-23 --override env.GIT_PYTHON_REFRESH="quiet" --override env.GOOGLE_APPLICATION_CREDENTIALS="/google/.google-secret-key.json" --override env.CHROMIUM_FLAGS="--no-sandbox"
If I manually delete the infra_overrides-block from the deployment-yaml - the flow works again. So somehow this env.xyz - overrides are the problem. Can you point me to my mistake?
Edit: I tested overriding the "image" - this works. It really only fails when overriding the environment variables....Sachin Ranveer
08/17/2022, 11:53 AMJason Thomas
08/17/2022, 12:33 PM2.0.4
Hi all, I just upgraded from 2.0b8
to 2.0.4
.
In v2.0b8
errors within flows/tasks were caught by Prefect and the flow would continue, now in 2.0.4
they are being raised and crashing my run. How can I get back to the previous behavior?
Iโll post my code in the threadPedro Machado
08/17/2022, 12:39 PMHanno Kรผpers
08/17/2022, 12:59 PMstate.result()
only returns _Result(key='82ecff4d3a454e77b08b5ce93e1dd418', filesystem_document_id=UUID('af2119c0-c631-45ed-af6a-0cff1cf396a5'))
. I tried different return value types for the flow (str, dict). What am I missing here? How can I easily access the return value of a succesful flow run? Thank youMars
08/17/2022, 2:03 PMHamza Naanani
08/17/2022, 2:05 PMcloud login
command ?
I tried setting prefect config set PREFECT_API_URL
and PREFECT_API_KEY
as a setting but it doesn't seem to work.Alberto Fernรกndez
08/17/2022, 2:12 PMTom Klein
08/17/2022, 3:04 PMMapped
- with no mapped tasks being created, and then to the flow itself just immediatley moving to the Failed
stated - even though nothing failed?Darren
08/17/2022, 3:08 PMssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)
my credentials look like this email_server_credentials = EmailServerCredentials(
username="demo@company.com",
password="mypasswordhere",
smtp_server="smtp.office365.com",
smtp_type="STARTTLS",
smtp_port=587
)Neil Natarajan
08/17/2022, 3:26 PMNot Ready
with the future.result()
resolving to None
. The issue is that two of the final tasks of my flow accept futures of previous tasks as input and are listed in the wait_for
section as well. These final two tasks instead are stuck in a Pending
state and this causes the flow to hang instead of running to completion. Any advice on how to properly implement the ideal behavior (tasks that depend on futures from previous tasks that fail or didn't run should also fail/not run and flow should run to completion) would be greatly appreciated, thank you!
In the logs, I see the flow encountered an UpstreamTaskError
.Ben Strange
08/17/2022, 3:29 PMSam Garvis
08/17/2022, 4:20 PMLucien Fregosi
08/17/2022, 4:38 PMKubernetesJob
is it possible to specify a node-selector
and tolerations
to be able to assign the worker pod to a specific node pool ?
I canโt see it in the doc
Thanks for your helpJeffrey Lam
08/17/2022, 4:48 PMJeffrey Lam
08/17/2022, 4:48 PMOscar Bjรถrhn
08/17/2022, 5:18 PMJeffrey Lam
08/17/2022, 5:51 PMOscar Bjรถrhn
08/17/2022, 5:57 PM