Gitanshu Sardana
08/11/2020, 10:28 PMGitanshu Sardana
08/11/2020, 10:28 PMalex
08/11/2020, 10:42 PMslack_notifier
as my flow's state_handler, but it does not seem to get called when my task fails. I get notified when it starts and ends.
My pipeline is set such that failing_task -> task (trigger_will_fail) -> recovery_task (task triggers only on fail and sets state to success, otherwise is skipped). I ideally want to be notified that we encountered a failure, even though we do end up doing failure recovery and setting the final state to a success.Riley Hun
08/11/2020, 11:39 PMfrom prefect import Flow
my_flow = Flow.load('thinknum-etl')
my_flow.run()
qizzle
08/12/2020, 3:33 AMqizzle
08/12/2020, 3:34 AMAvi A
08/12/2020, 9:09 AMEthan Shenker
08/12/2020, 2:33 PMfinished task run for task with final state: 'Paused'
, and I'm not sure as to how this would be fixed.Hannah Amundson
08/12/2020, 4:12 PMLuke Orland
08/12/2020, 4:31 PMpip install prefect[aws,dask_cloudprovider]
currently fails to install (without some extra work) due to conflicting dependencies.
A solution I found was:
pip install --use-feature=2020-resolver prefect[aws,dask_cloudprovider]
If you pin versions in requirements.txt or use pipenv or pip-tools, you can then run
$ pip freeze | grep boto
aiobotocore==1.0.7
boto3==1.12.32
botocore==1.15.32
to see the versions you need to pin to.Riley Hun
08/12/2020, 4:33 PMprefecthq/prefect:0.12.1-python3.7
and prefecthq/prefect:0.12.5-python3.7
, which resulted in the following error:
AttributeError: module 'types' has no attribute 'CellType'
I've also tried FROM prefecthq/prefect:0.11.5-python3.7
, which resulted in this error:
ModuleNotFoundError: No module named 'prefect.core.parameter'
alex
08/12/2020, 5:28 PMCan't connect to <http://localhost:4200/graphql>
error when running my server on a remote instance. I've upgraded my prefect version so that might be why.
Here's my config.toml
[server]
[server.ui]
graphql_url = "<http://url:4200/graphql>"
[server.api]
graphql_url = "<http://url:4200>"
Here's my output for prefect diagnostics
{
"config_overrides": {
"backend": true,
"graphql_url": true,
"server": {
"api": {
"SLACK_WEBHOOK_URL": true,
"graphql_url": true
},
"ui": {
"graphql_url": true
}
}
},
"env_vars": [],
"system_information": {
"platform": "Linux-5.3.0-1032-aws-x86_64-with-Ubuntu-18.04-bionic",
"prefect_version": "0.13.2",
"python_version": "3.6.8"
}
}
Riley Hun
08/12/2020, 6:41 PMValueError: Task <Task: ThinkNumGetHistoryTask> has retry settings but some upstream dependencies do not have result types
Would anyone know how to address this error? I tried adding retry to all the tasks in the flow, but didn't seem to resolve the issue.Riley Hun
08/12/2020, 8:12 PMModuleNotFoundError: No module named 'alternative_data_pipelines'
Note that the package alternative_data_pipelines
exists in my docker file and has been pip-installed. No issues with the health checker - everything has been registered successfully and runs ok locally and from the docker container.Slackbot
08/12/2020, 8:14 PMitay livni
08/12/2020, 9:59 PMflatten
filter out None
by default?Jacob Blanco
08/13/2020, 1:59 AMrun
on the parameter in the flow definition is clearly wrong.
The Parameter instance/s are passed in as a list since the users of the Task need to be able to pass an arbitrary number of unnamed parameters. Are there any examples of unpacking the parameters in an efficient way?
So far I have
params = [param.run() if isinstance(param, Parameter) else param for param in params]
I've also tried running the Parameter through a @task
function but when I pass the results of that task to DoSomething it complains that FunctionalTask is not compatible.Robin Cole
08/13/2020, 8:10 AMJulian
08/13/2020, 11:26 AMapply_map
?
My problem is, that when I map more than 60 Tasks at once, each having an open connection to an external database, the flow will remain in a running state and never terminate.Aditya Mandke
08/13/2020, 12:43 PMHugues-Yanis Amanieu
08/13/2020, 3:01 PMHugues-Yanis Amanieu
08/13/2020, 3:09 PMMarwan Sarieddine
08/13/2020, 11:12 PM78s Warning Evicted pod/dask-worker-855c59c78d-qwv5v The node had condition: [DiskPressure].
78s Warning Evicted pod/dask-worker-855c59c78d-fvc4q The node had condition: [DiskPressure].
I am wondering if someone faced similar issues ?Mikael
08/14/2020, 7:13 AMMarwan Sarieddine
08/14/2020, 12:35 PMAdam
08/14/2020, 1:53 PMBigQueryTask
on Prefect Cloud and it failed with the error “No heartbeat detected from the remote task; marking the run as failed”. Was this Zombie killer? It worked fine locally. How can I ensure this long-running task completes?bral
08/14/2020, 2:09 PMWill Milner
08/14/2020, 2:40 PM[server]
[server.ui]
graphql_url = "<http://10.0.0.300:4200/graphql>"
But when I connect the UI still attempts to connect to localhost:4200/graphql
Riley Hun
08/14/2020, 4:31 PMbral
08/14/2020, 7:08 PM