@Marvin have you seen this error before? it started happening all of a sudden

18:18:17.506 | ERROR   | Task run 'get_db_manager-0' - Crash detected! Execution was interrupted by an unexpected exception: ValueError: ctypes objects containing pointers cannot be pickled
18:18:17.666 | ERROR   | Flow run 'cryptic-swan' - Encountered exception during execution:
Traceback (most recent call last):
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/engine.py", line 877, in orchestrate_flow_run
    result = await flow_call.aresult()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult
    return await asyncio.wrap_future(self.future)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/Cambium/data-canopy/data-flows/dataflows/flows/projects_observability_export_flow.py", line 228, in projects_observability_export_flow
    db_manager = get_db_manager(connection_params)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/tasks.py", line 689, in __call__
    return enter_task_run_engine(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/engine.py", line 1421, in enter_task_run_engine
    return from_sync.wait_for_call_in_loop_thread(begin_run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/api.py", line 218, in wait_for_call_in_loop_thread
    return call.result()
           ^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/engine.py", line 1555, in get_task_call_return_value
    return await future._result()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/states.py", line 91, in _get_state_result
    raise await get_state_exception(state)
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/task_runners.py", line 231, in submit
    result = await call()
             ^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/engine.py", line 1806, in begin_task_run
    state = await orchestrate_task_run(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/engine.py", line 2130, in orchestrate_task_run
    terminal_state = await return_value_to_state(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/states.py", line 305, in return_value_to_state
    return Completed(data=await result_factory.create_result(data))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/results.py", line 453, in create_result
    return await PersistedResult.create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/results.py", line 681, in create
    data = serializer.dumps(obj)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/serializers.py", line 148, in dumps
    blob = pickler.dumps(obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)

Your message was too long by 1071 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:

have you seen this error before? it started happening all of a sudden

181817.506 | ERROR | Task run 'get_db_manager-0' - Crash detected! Execution was interrupted by an unexpected exception: ValueError: ctypes objects containing pointers cannot be pickled 181817.666 | ERROR | Flow run 'cryptic-swan' - Encountered exception during execution: Traceback (most recent call last): File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/engine.py", line 877, in orchestrate_flow_run result = await flow_call.aresult() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 327, in aresult return await asyncio.wrap_future(self.future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mmoncadaisla/anaconda3/envs/data_env/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 352, in _run_sync result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/mmoncadaisla/Cambium/data-canopy/data-flows/dataflows/flows/projects_ob

@Marvin have you seen this error before? it started happening all of a sudden:

18:18:17.506 | ERROR   | Task run 'get_db_manager-0' - Crash detected! Execution was interrupted by an unexpected exception: ValueError: ctypes objects containing pointers cannot be pickled

The error message

ctypes objects containing pointers cannot be pickled

typically arises because the ctypes module in Python involves C-level pointer manipulations that aren't serializable by the default pickle processes. From the Prefect documentation, serializing objects with complex pointers and references can be problematic: - Pickle Serializer: Prefect uses

cloudpickle

by default to serialize complex Python objects, but it’s hard to pinpoint the cause when nested attributes (like pointers) can't be pickled. - link: Pickle Serializer - Prefect Docs In this case, you can try to solve this issue by either avoiding the use of such ctypes objects in your Prefect tasks or by designing your workflow to handle these objects differently: 1. Avoid ctypes objects or redesign data flow: If possible, refactor your code to avoid passing ctypes objects through Prefect tasks. 2. Custom serialization: Implement custom serialization methods if you must use these objects, perhaps converting pointers to a format that can be serialized. 3. Alternative data transfer methods: Use alternative data transfer methods (e.g., database storage, file systems) to pass these objects outside normal Prefect task execution. Unfortunately, there are no relevant GitHub issues found that directly address this error, which means it might be a relatively less encountered or reported problem in the Prefect GitHub repository. For more detailed assistance, you might consider providing a minimal reproducible example or consulting Prefect Discourse where the community or Prefect developers can provide insights based on more context. Is there any other way I can assist you with this issue, or is there something specific you'd like to explore within this context?

i suspect you're trying to serialize some (unserializable) db connection object of some kind? just based on > 'get_db_manager-0'

Yes, I have removed these tasks that return object instances