Ayyanar Thangaraj
06/16/2021, 7:21 AMThomas Hoeck
06/16/2021, 7:23 AMFailed to set task state with error: ClientError([{'path': ['set_task_run_states'], 'message': '[{\'extensions\': {\'path\': \'$.selectionSet.billing_license.selectionSet.account_name\', \'code\': \'validation-failed\'}, \'message\': \'field "account_name" not found in type: \\\'billing_license\\\'\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}])
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/cloud/task_runner.py", line 98, in call_runner_target_handlers
state = self.client.set_task_run_state(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1503, in set_task_run_state
result = self.graphql(
File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 319, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['set_task_run_states'], 'message': '[{\'extensions\': {\'path\': \'$.selectionSet.billing_license.selectionSet.account_name\', \'code\': \'validation-failed\'}, \'message\': \'field "account_name" not found in type: \\\'billing_license\\\'\'}]', 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
I'm running Prefect 0.14.7 in the flows and the agent is 0.14.17Ayyanar Thangaraj
06/16/2021, 7:24 AMJeremy Phelps
06/16/2021, 10:27 AMKeyError: 'Task slug disable_sound_on_mixer_error_stores-1 not found in the current Flow; this is usually caused by changing the Flow without reregistering it with the Prefect API.'
The slug is a transformation of the name of a flow. Prefect replaced the hyphens with _
and added -1
to the end.
The error is happening when trying to run a different flow. Both flows are defined in the same Python source file.
Both flows were registered with the latest code, contrary to the suggestion in the error message.João Amorim
06/16/2021, 11:56 AMBrad I
06/16/2021, 3:43 PMAlexander
06/16/2021, 4:51 PMAn error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.
because generated definition lacks requiresCompatibilities
field. Trying to solve that with custom task definition, i constantly run into different errors like missing field here and there and a problem that definition registration script patches my task definition with its invalid values. Right now i am unable to fight a problem when generated definition have null container name, though docker storage, and run config have them set. I can not debug anything without any logging.jcozar
06/16/2021, 5:27 PMPaul Schale
06/16/2021, 6:59 PMtyping.NewType()
for a task type hint, the idempotency key becomes non-deterministic. From what I can tell, it’s because typing.NewType()
actually returns a function, and calling repr
or str
on a function includes the memory address in the result. Here is a simple flow that reproduces the issue:
import prefect
import typing
MyNewType = typing.NewType("MyNewType", str)
@prefect.task(name="print_something")
def print_something(input_str: MyNewType):
print(input_str)
with prefect.Flow("my_flow") as flow:
print_something("test_str")
# This is different every time script is run
print(flow.serialized_hash())
# Output includes memory address, e.g.
# "<function NewType.<locals>.new_type at 0x7fa6c71c34c0>"
print(list(flow.tasks)[0].inputs()["input_str"]["type"]
As for fixing this, I don’t know enough about the serialization process to know where to start. It can be mitigated by wrapping the type hint in typing.Optional[]
.
I’m running prefect version 0.14.22 on python 3.9.1Hugo Kitano
06/16/2021, 9:42 PMERROR - proteograph-ecs-agent | Failed to infer default networkConfiguration, please explicitly configure using
--run-task-kwargs`` with the most recent prefect version. I’ve run it successfully with other ec2 instances so i’m not sure what’s going onEvan Crook
06/16/2021, 10:47 PMextract -> map over might_fail -> map over show
and any failed mapped might_fail
instance passes its result to handle_error
. might_fail
also includes retries.
However what I'm seeing is: the task might_fail[3]
starts its task run, hits a RuntimeError
, enters a Retrying
state, and then finishes its task run with state Retrying
. It does not do any retries. Then the rest of the mapped tasks finish and we proceed to `handle_failure`; and handle_failure[3]
says "Not all upstream states are finished; ending run."
(because the upstream state is Retrying
, not Failed
) and then Finished task run for task with final state: 'Pending'
. So handle_failure
never actually runs at all.
If I take out retries on might_fail
it works as expected. But in the real-world example this is mimicking, might_fail
hits an API prone to rate limits / transient failures so we actually want to retry it, and then trigger handle_failure
, only if it's entered a failed state after retrying a few times.
Does this make sense? Is this a bug or am I just doing something terribly wrong? (happy to provide logs from running this minimal example, too, if it's helpful)
thanks so much in advance! 🌟Sumit Kumar Rai
06/17/2021, 3:39 AMfrom prefect.client import Secret
GITHUB_ACCESS_TOKEN = Secret("GITHUB_ACCESS_TOKEN").get()
g.suijker
06/17/2021, 7:35 AME: Failed to fetch <https://packages.microsoft.com/debian/10/prod/pool/main/u/unixodbc/odbcinst_2.3.7_amd64.deb> 404 Not Found [IP: 104.214.230.139 443]
E: Failed to fetch <https://packages.microsoft.com/debian/10/prod/pool/main/u/unixodbc/unixodbc-dev_2.3.7_amd64.deb> 404 Not Found [IP: 104.214.230.139 443]
E: Failed to fetch <https://packages.microsoft.com/debian/10/prod/pool/main/u/unixodbc/odbcinst1debian2_2.3.7_amd64.deb> 404 Not Found [IP: 104.214.230.139 443]
E: Failed to fetch <https://packages.microsoft.com/debian/10/prod/pool/main/u/unixodbc/libodbc1_2.3.7_amd64.deb> 404 Not Found [IP: 104.214.230.139 443]
E: Failed to fetch <https://packages.microsoft.com/debian/10/prod/pool/main/u/unixodbc/unixodbc_2.3.7_amd64.deb> 404 Not Found [IP: 104.214.230.139 443]
E: Failed to fetch <https://packages.microsoft.com/debian/10/prod/pool/main/m/msodbcsql17/msodbcsql17_17.7.2.1-1_amd64.deb> 404 Not Found [IP: 104.214.230.139 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
I'm using a custom docker file as provided here https://docs.prefect.io/orchestration/recipes/configuring_storage.html:
FROM prefecthq/prefect:0.14.10-python3.8
# install some base utilities
RUN apt update && apt install build-essential -y build-essential unixodbc-dev && rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install curl -y
# install mssql-tools
RUN curl <https://packages.microsoft.com/keys/microsoft.asc> | apt-key add -
RUN curl <https://packages.microsoft.com/config/debian/10/prod.list> > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update && ACCEPT_EULA=Y apt-get install msodbcsql17 -y
RUN ACCEPT_EULA=Y apt-get install mssql-tools -y
# update bash configuration
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
# update OpenSSL configuration file
RUN sed -i 's/TLSv1\.2/TLSv1.0/g' /etc/ssl/openssl.cnf
RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/g' /etc/ssl/openssl.cnf
Sumit Kumar Rai
06/17/2021, 7:51 AMpipelinewise
command as a shell task in a prefect flow. The command takes config files and a state file as an input parameter. The command executes and updates the state file at the end. What is the best way to retain the state file and also read the latest state file before executing the task?Florian Kühnlenz
06/17/2021, 8:17 AMStéphan Taljaard
06/17/2021, 9:27 AMshared_credentials
with contents {"sendgrid-api-key": "abc", "some-other-api-key": "efg", ...}
,
Is there a way to read the sendgrid-api-key
value from shared_credentials
, then create a temporary secret in the flow named SENDGRID_API_KEY
, as required by the SendGrid task?
Or other ideas to use the value inside of the secret?Domantas
06/17/2021, 10:36 AMTypeError: cannot pickle '_thread.lock' object
(I'll paste full error code in the comments) and I'm out of ideas how to properly solve it.
It is related with DaskExecutor
and it appears when trying to proceed this task(I'll upload code sample in the comments):
1. Read file content from a yml file which is located in the S3 storage. For file reading I'm using raw boto3
implementation.
2. Read bytes from the downloded yml file
3. Load yml file and convert it into list
Does anyone knows a solution to this problem?Ben Muller
06/17/2021, 11:14 AMciaran
06/17/2021, 1:54 PMpod_template
and scheduler_pod_template
in my dask_kubernetes.KubeCluster
DaskExecutor
.
Both the templates (for now) are exactly the same:
"pod_template": make_pod_spec(
image=os.environ["BAKERY_IMAGE"],
labels={"flow": flow_name},
env={
"AZURE_STORAGE_CONNECTION_STRING": os.environ[
"FLOW_STORAGE_CONNECTION_STRING"
]
},
),
"scheduler_pod_template": make_pod_spec(
image=os.environ["BAKERY_IMAGE"],
labels={"flow": flow_name},
env={
"AZURE_STORAGE_CONNECTION_STRING": os.environ[
"FLOW_STORAGE_CONNECTION_STRING"
]
},
),
If I try to run a flow with both declared, my Dask Scheduler pod fails with:
Traceback (most recent call last): File "/srv/conda/envs/notebook/bin/dask-worker", line 8, in <module>
sys.exit(go())
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 462, in go
main()
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 406, in main
nannies = [
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 407, in <listcomp>
t(
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/nanny.py", line 220, in __init__
host = get_ip(get_address_host(self.scheduler.address))
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/comm/addressing.py", line 142, in get_address_host
return backend.get_address_host(loc)
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/comm/tcp.py", line 572, in get_address_host
return parse_host_port(loc)[0]
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/comm/addressing.py", line 90, in parse_host_port
port = _default()
File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/comm/addressing.py", line 69, in _default
raise ValueError("missing port number in address %r" % (address,))
ValueError: missing port number in address '$(DASK_SCHEDULER_ADDRESS)'
But if I only declare the pod_template
, everything works out great. I'm assuming the fact I declare the scheduler_pod_template
means I'm losing some default setup somewhere down the line?Joe
06/17/2021, 2:17 PMFailed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
liren zhang
06/17/2021, 3:35 PMStartFlowRun
to invoke individual flows in their defined orders. All the flows run in their own docker container. When master flow execution flow fails due to specific failed execution, it was not working. I read in docs and said that docker run does not yet support retaining results/inputs/outputs.
I read a thread here by @Kevin Kho and seemed to have suggested that PrefectResult may be a workaround if we don't care about the data we push to Prefect Cloud.
https://prefect-community.slack.com/archives/CL09KU1K7/p1622645858327700
I am not exactly sure how PrefectResult can be used in our scenario. Please help!Zach Schumacher
06/17/2021, 3:37 PMJonathan Pou
06/17/2021, 3:48 PMRiley Hun
06/17/2021, 5:09 PMBen Muller
06/18/2021, 6:14 AMBen Muller
06/18/2021, 6:14 AM--profile staging
Ben Muller
06/18/2021, 7:00 AMRustam Iskenderov
06/18/2021, 10:00 AM[2021-06-18 12:33:54+0300] INFO - prefect.FlowRunner | Beginning Flow run for 'download'
[2021-06-18 12:33:54+0300] INFO - prefect.TaskRunner | Task 'check_file': Starting task run...
Checking file availability plip ...
[2021-06-18 12:33:54+0300] INFO - prefect.TaskRunner | Task 'check_file': Finished task run for task with final state: 'Success'
[2021-06-18 12:33:54+0300] INFO - prefect.TaskRunner | Task 'download_file': Starting task run...
Downloading file plip ...
[2021-06-18 12:33:54+0300] INFO - prefect.TaskRunner | Task 'download_file': Finished task run for task with final state: 'Success'
[2021-06-18 12:33:54+0300] INFO - prefect.TaskRunner | Task 'Flow process': Starting task run...
[2021-06-18 12:33:55+0300] INFO - prefect.TaskRunner | Task 'Flow process': Finished task run for task with final state: 'Success'
[2021-06-18 12:33:55+0300] INFO - prefect.FlowRunner | Flow run SUCCESS: all reference tasks succeeded
jcozar
06/18/2021, 10:01 AMMichael Law
06/18/2021, 10:04 AMMichael Law
06/18/2021, 10:04 AMStéphan Taljaard
06/18/2021, 10:17 AMflow.register(
project_name=flow_build.project,
idempotency_key=flow.serialized_hash(),
)
Michael Law
06/18/2021, 10:22 AM