I am trying to run a simple task, but I am getting...
# ask-communityt
Tadej Svetina
10/08/2021, 5:26 PMI am trying to run a simple task, but I am getting
Unexpected error: TypeError("cannot pickle '_io.BufferedReader' object")Tadej Svetina
10/08/2021, 5:27 PMHere's the flow
Copy code
with Flow("Hello", schedule) as flow:
h = say_hello()
t1 = LambdaInvoke("test_prefect", log_type="Tail")
t1(upstream_tasks=[h])
t2 = LambdaInvoke("test_prefect", log_type="Tail")
t2(upstream_tasks=[h])
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=2)Tadej Svetina
10/08/2021, 5:27 PMAnd the full call stack (running with local agent)
Copy code
Unexpected error: TypeError("cannot pickle '_io.BufferedReader' object")
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 909, in get_task_run_state
result = self.result.write(value, **formatting_kwargs)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/results/local_result.py", line 115, in write
value = self.serializer.serialize(new.value)
File "/usr/local/lib/python3.9/site-packages/prefect/engine/serializers.py", line 73, in serialize
return cloudpickle.dumps(value)
File "/usr/local/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/usr/local/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 602, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle '_io.BufferedReader' objectk
Kevin Kho
10/08/2021, 5:32 PM
Do you have a task that returns a boto3 client? Prefect requires task outputs to be serializable. You can read more on this here.
t
Tadej Svetina
10/08/2021, 5:34 PMThis is a task from the prefect library
https://docs.prefect.io/api/latest/tasks/aws.html#lambdainvoke
Tadej Svetina
10/08/2021, 5:36 PMOk I see, prefect just directly returns the response, without taking our
StreamingBodyk
Kevin Kho
10/08/2021, 5:39 PM
None of these task library tasks are official. 80-85% of them are community contributed. This task is used by a bunch of users though. If it doesn’t fit your use case, you can subclass it or copy the code over and make your own tasks.
t
Tadej Svetina
10/08/2021, 5:43 PMI see. It's really hard to know these are community contributed tasks (it's right up there in the API documentation with cli, client, engine). I think it would be better if you properly separate this, or raise the standards if you want to have it so tightly integrated
Tadej Svetina
10/08/2021, 5:44 PMIt's not that it doesn't fit my use case, it doesn't pass basic checks. This task probably only works in local execution, so for all intents and purposes it is broken
k
Kevin Kho
10/08/2021, 5:46 PM
That’s fair, a warning in the docs should be there.
👍 1
Kevin Kho
10/08/2021, 5:47 PM
@Marvin open “Task Library Needs a Warning that It Is Community Contributed”
m
Marvin
10/08/2021, 5:47 PM