Michael Reynolds
07/16/2022, 11:24 PMAnna Geller
07/17/2022, 10:57 AM23:29:26.313 | ERROR | Flow run 'mustard-quetzal' - Encountered exception during execution:
Traceback (most recent call last):
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/engine.py", line 521, in orchestrate_flow_run
flow_call
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/utilities/asyncio.py", line 120, in run_sync_in_interruptible_worker_thread
cancellable=True,
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
raise exceptions[0]
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 702, in _run_wrapped_task
await coro
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/to_thread.py", line 32, in run_sync
func, *args, cancellable=cancellable, limiter=limiter
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/utilities/asyncio.py", line 96, in capture_worker_thread_and_result
result = __fn(*args, **kwargs)
File "/home/myapp/.local/lib/python3.7/site-packages/myapp/main.py", line 35, in run_pipeline
messages = poll_kafka( conf[ 'kafka' ] ).result()
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/futures.py", line 210, in result
self._result, timeout=timeout, raise_on_failure=raise_on_failure
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/utilities/asyncio.py", line 221, in sync
return run_async_from_worker_thread(__async_fn, *args, **kwargs)
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/utilities/asyncio.py", line 136, in run_async_from_worker_thread
return anyio.from_thread.run(call)
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/from_thread.py", line 49, in run
return asynclib.run_async_from_thread(func, *args)
File "/home/myapp/.local/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
return f.result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/futures.py", line 220, in _result
return final_state.result(raise_on_failure=raise_on_failure)
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/orion/schemas/states.py", line 136, in result
raise data
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/task_runners.py", line 314, in _run_and_store_result
self._results[run_key] = await run_fn(**run_kwargs)
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/engine.py", line 799, in begin_task_run
client=client,
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/engine.py", line 905, in orchestrate_task_run
result, serializer="cloudpickle"
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/states.py", line 130, in return_value_to_state
return Completed(data=DataDocument.encode(serializer, result))
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/orion/schemas/data.py", line 42, in encode
blob = lookup_serializer(encoding).dumps(data, **kwargs)
File "/home/myapp/.local/lib/python3.7/site-packages/prefect/serializers.py", line 59, in dumps
data_bytes = cloudpickle.dumps(data)
File "/home/myapp/.local/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/home/myapp/.local/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 633, in dump
return Pickler.dump(self, obj)
TypeError: can't pickle cimpl.Message objects
cimpl.Message
objects to the task that needs it without passing it between tasks?
checkpoint=False is a good approach but this doesn't exist in 2.0 yetMichael Reynolds
07/18/2022, 3:31 PMcheckpoint = False
is the correct way to go. is there an issue for adding that support that i could follow?Anna Geller
07/18/2022, 4:05 PM