Issam Assafi
09/23/2021, 12:38 PMSteve s
09/23/2021, 1:03 PMDmitry Kuleshov
09/23/2021, 1:18 PMAnze Kravanja
09/23/2021, 2:49 PMSean Talia
09/23/2021, 8:14 PMDockerAgent
, and for some reason, the agent seems to keep a flow's container running long after the flow actually finished?Kathryn Klarich
09/23/2021, 8:29 PM*prefect.client.client.Client.write_run_logs*(logs)
- but i want to download, not upload the logsDaniel Saxton
09/23/2021, 9:17 PMprefect run
is there anything like a -d
flag (as in say docker run
) to detach the process and run in the background (i didn't see it in the docs but wanted to double check)?Daniel Saxton
09/23/2021, 9:17 PMFabrice Toussaint
09/24/2021, 10:35 AMFailed to set task state with error: ClientError([{'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.'}}}])
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 91, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 1917, in set_task_run_state
result = self.graphql(
File "/opt/conda/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.', 'locations': [{'line': 2, 'column': 5}], 'path': ['set_task_run_states'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'State update failed for task run ID 6aa4d54f-d2a5-41d6-9ba8-27ecffe5e209: provided a running state but associated flow run ecb87527-e50d-4ca4-9ed6-4c2f40ef81fe is not in a running state.'}}}]
We are using Prefect 0.15.2 and have a Dask cluster up and running.
Also this, we are receiving multiple apollo timeouts but on a very random basis:
Failed to set task state with error: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='apollo', port=4200): Read timed out. (read timeout=15)"))
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/opt/conda/lib/python3.8/http/client.py", line 1348, in getresponse
response.begin()
File "/opt/conda/lib/python3.8/http/client.py", line 316, in begin
version, status, reason = self._read_status()
File "/opt/conda/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/conda/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
Kevin Weiler
09/24/2021, 2:14 PMstate_handler
? It’s sort of surprising to me that, for example, the kubernetes RunNamespaceJob
doesn’t seem to include any logic around this (https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/kubernetes/job.py#L580).Lucas Beck
09/24/2021, 2:24 PMBruno Murino
09/24/2021, 4:24 PMFrancis Addae
09/24/2021, 4:56 PMKen Nguyen
09/24/2021, 5:42 PMdef post_to_slack(obj, old_state, new_state):
# Notify failures
if new_state.is_failed():
msg = "Flow/Task {0} finished in state {1}".format(obj, new_state)
secret_slack = fail_channel_webhook
<http://requests.post|requests.post>(secret_slack, json={"text": msg})
# Notify runs
if new_state.is_running():
msg = "Flow/Task {0} is running".format(obj)
secret_slack = running_channel_webhook
<http://requests.post|requests.post>(secret_slack, json={"text": msg})
return new_state
When I tested it on tasks, I was able to receive notifications for task running, and then task failing. However, when I tested it on flows, it only notified me of flow running, but not failing despite the flow's end result being Flow run FAILED: some reference tasks failed.
What differs between the flow and the task for this to not have worked? Thanks in advance for your suggestions!Irakli Gugushvili
09/24/2021, 5:47 PMwarmwaffles
09/24/2021, 7:08 PMwarmwaffles
09/24/2021, 8:43 PMwarmwaffles
09/24/2021, 9:57 PMAaron Ash
09/25/2021, 12:46 AMWilliam Grim
09/25/2021, 3:41 AMprefect get flow-runs -f myflow
. We also see a few runs with Success
, Failed
, and Cancelled
as well. I don't know how someone got a flow to cancel though.Abhishek
09/25/2021, 2:08 PMNo heartbeat detected from the remote task; marking the run as failed.
Other details:
• Its running on ECS (agent is ECS)
• Flow storage is S3
• In the same Flow other long running tasks did run successfully
What can i do to prevent this timeouts? and how to fix it. As this doc suggests Lazarus will restart the failed task.
but i need a way to prevent the timeout if possible.
Screenhot:Guy Propper
09/26/2021, 10:56 AMfile_paths_to_process_list = extract_files_to_process()
mapped_parsed_output = transform.map(file_paths_to_process_list)
load_output_to_db.map(mapped_parsed_output)
I want map#2 (load_output_to_db) to depend on results from map#1 (transform), but not to wait for all results from map#1. As soon as there is one result from map#1, it should be processed in map#2.
What is the correct way to do this?
Thanks!Henning Holgersen
09/26/2021, 11:14 AMAmar Verma
09/26/2021, 3:29 PMLew Dawson
09/26/2021, 7:37 PMDockerRun
run_config with Github
storage. Have the following project structure in the Github repo:
etl/
├── util/
│ ├── __init__.py
│ ├── util_vault.py
├── __init__.py
├── runner.py
The util_vault.py
looks like:
from prefect.client import Secret
class UtilVault:
@staticmethod
def get_secret_simple(name):
return Secret(name).get()
The runner.py
looks like:
from prefect import Flow, task, context
from prefect.run_configs import DockerRun
from prefect.storage import GitHub
from util.util_vault import UtilVault
logger = context.get("logger")
@task
def hello_secrets():
secret = UtilVault.get_secret_simple('GITHUB_ACCESS_TOKEN')
<http://logger.info|logger.info>('secret = %s', secret)
with Flow(name='hello_world') as flow:
hello_secrets()
flow.storage = GitHub(
repo='<username>/<repo>',
path='etl/runner.py',
access_token_secret='GITHUB_ACCESS_TOKEN',
)
flow.run_config = run_config=DockerRun(
image='<registry-name>/prefect-runner-base-image:1.0.0',
)
When I run this task, I get ModuleNotFoundError: No module named 'util'
. Any help would be much appreciated.Joël Luijmes
09/27/2021, 9:44 AMFailed to retrieve task state with error: ClientError([{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 154, in initialize_run
task_run_info = self.client.get_task_run_info(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 1798, in get_task_run_info
result = self.graphql(mutation) # type: Any
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': 'Expected type UUID!, found ""; Could not parse UUID: ', 'locations': [{'line': 2, 'column': 5}], 'path': ['get_or_create_task_run_info'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': 'Expected type UUID!, found ""; Could not parse UUID: '}}}]
I found the following issue: https://github.com/PrefectHQ/prefect/issues/4687, but it says the root issue should be fixed. Note; this happens for all of my flows (of which all are relative small amount of tasks).Sergey
09/27/2021, 12:45 PMKevin Kho
09/27/2021, 1:52 PMSam Thomas
09/27/2021, 3:02 PMTask 'run_edit_connection_graph_flow[1]': Exception encountered during task execution!
Traceback (most recent call last):
File "ENV\lib\site-packages\prefect\engine\task_runner.py", line 859, in get_task_run_state
value = prefect.utilities.executors.run_task_with_timeout(
File "ENV\lib\site-packages\prefect\utilities\executors.py", line 454, in run_task_with_timeout
return task.run(*args, **kwargs) # type: ignore
File "MY_DRIVE\tasks.py", line 688, in run_X_flow
prefect.tasks.prefect.StartFlowRun(
File "ENV\lib\site-packages\prefect\utilities\tasks.py", line 445, in method
return run_method(self, *args, **kwargs)
File "ENV\lib\site-packages\prefect\tasks\prefect\flow_run.py", line 432, in run
create_link(urlparse(run_link).path)
File "ENV\lib\site-packages\prefect\artifacts.py", line 52, in create_link
return _create_task_run_artifact("link", {"link": link})
File "ENV\lib\site-packages\prefect\artifacts.py", line 28, in _create_task_run_artifact
return client.create_task_run_artifact(
File "ENV\lib\site-packages\prefect\client\client.py", line 2155, in create_task_run_artifact
result = self.graphql(
File "ENV\lib\site-packages\prefect\client\client.py", line 569, in graphql
raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'message': '[{\'extensions\': {\'path\': \'$.selectionSet.insert_task_run_artifact.args.objects\', \'code\': \'constraint-violation\'}, \'message\': \'Not-NULL violation. null value in column "tenant_id" violates not-null constraint\'}]', 'locations': [{'line': 2, 'column': 5}], 'path': ['create_task_run_artifact'], 'extensions': {'code': 'INTERNAL_SERVER_ERROR', 'exception': {'message': '[{\'extensions\': {\'path\': \'$.selectionSet.insert_task_run_artifact.args.objects\', \'code\': \'constraint-violation\'}, \'message\': \'Not-NULL violation. null value in column "tenant_id" violates not-null constraint\'}]'}}}]
Having issues after upgrading to 0.15.6 using StartFlowRun task.
Local executors pointing to a Prefect Server hosted on our infrastructure. The requested flow starts running however the flow that did the requesting fails when trying to create an artefact for the run it just started.
I'm surprised the issues is around tenants because my understanding was that they're Cloud things.
Any help gratefully recieved.Qin XIA
09/27/2021, 4:22 PMclean_up_task()
which is mapped to 180 files. It takes 30 seconds when I run this flow in local without agent and it takes 2 mins when i run it using local agent by UI. What can i do for optimizing this point. Thanks a lot.Qin XIA
09/27/2021, 4:22 PMclean_up_task()
which is mapped to 180 files. It takes 30 seconds when I run this flow in local without agent and it takes 2 mins when i run it using local agent by UI. What can i do for optimizing this point. Thanks a lot.Kevin Kho
09/27/2021, 4:26 PMQin XIA
09/27/2021, 4:30 PM