Joachim Zaspel
02/08/2021, 8:10 PMMatthew Blau
02/08/2021, 10:54 PMjeff n
02/08/2021, 11:28 PM@task
settings, and just sits waiting for an hour until the heartbeat dies. We can run it on the same worker server manually with no problems at all. There doesn’t seem to be any logs indicating it is out of memory. This seems pretty squarely on the prefect side but I am lost how to debug it since only the server has the issue.
Process 27956: /opt/prefect/.venv/bin/python3.8 -m prefect heartbeat flow-run -i 14303ee3-e96a-49c9-87e6-6577fcace7d3
Python v3.8.0 (/usr/bin/python3.8)
Thread 27956 (idle): "MainThread"
flow_run (prefect/cli/heartbeat.py:98)
invoke (click/core.py:610)
invoke (click/core.py:1066)
invoke (click/core.py:1259)
invoke (click/core.py:1259)
main (click/core.py:782)
__call__ (click/core.py:829)
<module> (prefect/__main__.py:4)
_run_code (runpy.py:85)
_run_module_as_main (runpy.py:192)
Milly gupta
02/08/2021, 11:31 PMGiovanni Giacco
02/08/2021, 11:50 PMflow.storage = Docker(registry_url="702*****<http://487.dkr.ecr.eu-central-1.amazonaws.com|487.dkr.ecr.eu-central-1.amazonaws.com>", image_tag='latest', python_dependencies=["dask_kubernetes"])
flow.run_config = KubernetesRun()
flow.executor = DaskExecutor(
cluster_class=lambda: KubeCluster(make_pod_spec(image=prefect.context.image)),
adapt_kwargs={"minimum": 3, "maximum": 5},
)
flow.register(project_name="AWS_Prefect_Kubernetes")
But I get the following error:
Unexpected error: AttributeError("can't set attribute")
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine/flow_runner.py", line 418, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/site-packages/prefect/executors/dask.py", line 211, in start
with self.cluster_class(**self.cluster_kwargs) as cluster: # type: ignore
File "C:/Users/giova/Git_Projects/PentaTechnology/earthalytics-engine/dask-flow-new.py", line 48, in <lambda>
File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/core.py", line 414, in __init__
super().__init__(**self.kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/spec.py", line 274, in __init__
super().__init__(
File "/usr/local/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 69, in __init__
self.name = str(uuid.uuid4())[:8]
AttributeError: can't set attribute
Any advice? It seems it's not be able to set the name of the pod into the cluster. I'm using Prefect 0.14.6.Sanjay Patel
02/09/2021, 12:37 AMMarwan Sarieddine
02/09/2021, 2:09 AMTask timed out during execution.
- this is quite unexpected given we don’t have any timeout set on the prefect task, has anyone else encountered this behavior before ?Riley Hun
02/09/2021, 6:16 AMMilly gupta
02/09/2021, 1:58 PMSean Talia
02/09/2021, 2:58 PMSecret("SECRET_KEY").get()
fails when I try to register my flow:
ValueError: Local Secret "SECRET_KEY" was not found.
my backend is set to cloud
, and I do not have use_local_secrets = true
set in my config.toml, so I'm not sure why prefect is trying to actually retrieve a local secret thereSamuel Hinton
02/09/2021, 4:20 PMPREFECT_SERVER__UI_HOST_PORT
. Our port 8080 is already in use by another service. Any ideas?Josh
02/09/2021, 4:31 PMrun
methods allowed to return arbitrary objects?
class ReturnObject:
def__init__(self, value: str = "hello", list: List = ["world"]):
self.value = value
self.list = list
class MyTask(Task):
def run(self) -> ReturnObject:
return ReturnObject("some value", ["a", "b", "c"])
task = MyTask()
with Flow() as flow:
result = task()
some_other_task(param1=result.value, param2=result.list)
If I try to run this code, I’ll get an error like
AttributeError: 'MyTask' object has no attribute 'value'
Adam Brusselback
02/09/2021, 5:45 PMAdam Brusselback
02/09/2021, 5:46 PMSamuel Hinton
02/09/2021, 6:01 PMprefect backend server --address=<http://something:4200/graphql>
?Vipul
02/09/2021, 6:26 PMMatthew Blau
02/09/2021, 7:42 PMAdam Brusselback
02/09/2021, 8:10 PMAdam Brusselback
02/09/2021, 8:10 PMMilly gupta
02/09/2021, 9:09 PMFina Silva-Santisteban
02/09/2021, 11:33 PMdockerfile
argument and adding a registry_url=MY_ECR_URL
param, but that doesn’t work and if I keep both params it tries to push the image to the ECR repo which I don’t want. What’s the right setup for this case?Kilian
02/10/2021, 8:16 AMSamuel Hinton
02/10/2021, 11:58 AMKIRYL BUCHA
02/10/2021, 12:51 PMSamuel Hinton
02/10/2021, 1:15 PMAWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
set doesnt seem to be what prefect is looking for), and it doesnt use them to authenticate. Does prefect only support public images?Cole Howard
02/10/2021, 3:01 PMAdam Brusselback
02/10/2021, 3:33 PMSamuel Hinton
02/10/2021, 4:08 PMMitchell Bregman
02/10/2021, 5:10 PMFailed
on the UI, while every task and the flow itself is working as expected.. what can be causing this?
[10 February 2021 12:06pm]: Unexpected error: TypeError('Could not serialize object of type Success.\nTraceback (most recent call last):\n File "/usr/local/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 49, in dumps\n result = pickle.dumps(x, **dump_kwargs)...
Rafał Wojdyła
02/10/2021, 5:16 PM