Hello I m encountering errors when passing a non JSON serial Prefect Community #ask-community

Hello! I'm encountering errors when passing a non-...

James Gatter

01/27/2023, 6:08 PM

Hello! I'm encountering errors when passing a non-JSON-serializable object, an AnnData, into my flows or from flow to subflow. If an object isn't serializable as JSON, does that mean it can't be passed to flows/subflows cloud deployments? Anyway, any solutions or workarounds are appreciated!

James Gatter

01/27/2023, 6:09 PM

__main___py.py

James Gatter

01/27/2023, 6:09 PM

Trace:

Copy code

ubuntu:~/Projects/$ python -m sandbox
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized

Current thread 0x00007f4132713740 (most recent call first):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/dtypes/generic.py", line 41 in _check
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/dtypes/generic.py", line 47 in _instancecheck
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/construction.py", line 444 in extract_array
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/arrays/_mixins.py", line 296 in __getitem__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1832 in getitem_block_index
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 4105 in _slice
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1638 in _get_slice_axis
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1602 in _getitem_axis
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1073 in __getitem__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/anndata/_core/anndata.py", line 336 in _init_as_view
  File "/home/ubuntu/.local/lib/python3.8/site-packages/anndata/_core/anndata.py", line 289 in __init__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/anndata/_core/anndata.py", line 1114 in __getitem__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 152 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  ...
Aborted (core dumped)

James Gatter

01/27/2023, 6:11 PM

Other info/details: • An AnnData is essentially composed of collections of pandas DataFrames, dictionaries, and a large numpy/scipy matrix (

.X

). It's stored in h5ad files which are based on the hdf5 file format (not plain text). • Using Prefect 2.7.8 • Using a local instance of Orion (--host 0.0.0.0) on AWS EC2 • I've tried not using Pydantic model as the input param and simply the AnnData itself and I get the same result

Zanie

01/27/2023, 7:05 PM

This looks like a bug in FastAPI’s

jsonable_encoder

which is used to convert flow parameters into a form that can be sent to the API

Zanie

01/27/2023, 7:06 PM

If it doesn’t cause a stack overflow, we can probably capture the error and note that this parameter is unserializable

Zanie

01/27/2023, 7:06 PM

However, this is quite the crash it doesn’t look like we can capture and ignore it

James Gatter

01/27/2023, 7:15 PM

thanks for the quick follow up Michael! what would be your suggested course of action?

Zanie

01/27/2023, 7:20 PM

I’d reproduce the stack overflow with just FastAPI’s jsonable encoder and open a ticket there, I’m curious if they’d fix it.

👍 1

Zanie

01/27/2023, 7:20 PM

I believe we do capture errors here, we allow unserializable parameters to be used. If they’re going to crash the whole Python process there’s not much we can do.

Zanie

01/27/2023, 7:21 PM

We could perhaps skip certain datatypes but that’s a little tricky

James Gatter

01/27/2023, 8:04 PM

Yeah,

jsonable_encoder(adata)

alone is enough to cause the error. https://github.com/tiangolo/fastapi/issues/4834 is essentially the issue I had without the stack overflow Based on the thread, if I post I don't think it'll be met with good reception 😅

James Gatter

01/27/2023, 8:08 PM

Only workaround I can think of is passing filepaths instead and having each workflow read the object into memory, but that's not as ideal when working on one machine. Is there any other way I can pass a non-serializable object to flows?

Zanie

01/27/2023, 9:00 PM

I think you could argue that crashing the process is far worse than raising an unclear exception

💯 1

Zanie

01/27/2023, 9:00 PM

Can you try annotating your parameter with

quote

when you pass it to the flow?

Zanie

01/27/2023, 9:01 PM

your_flow(foo=prefect.utilities.annotations.quote(your_param))

Zanie

01/27/2023, 9:01 PM

We may skip JSON serialization in that case, I’m not sure though.

Zanie

01/27/2023, 9:01 PM

What datatype is causing the problem?

James Gatter

01/27/2023, 9:01 PM

AnnData

James Gatter

01/27/2023, 9:02 PM

or do you mean exception wise? not sure because of the stack overflow

James Gatter

01/27/2023, 9:03 PM

I suppose I could try all the objects within the AnnData and find out which one as well! But yes for now I will try your suggestion! Thanks

Zanie

01/27/2023, 9:06 PM

I see I gotta look at AnnData 🙂

James Gatter

01/27/2023, 9:08 PM

Haha you should if you're interested in data science!

James Gatter

01/27/2023, 9:19 PM

Copy code

import prefect
from prefect import task, flow, get_run_logger
import scanpy as sc

def main():
    adata = sc.read_h5ad('demuxed.h5ad')
    mainflow(adata=prefect.utilities.annotations.quote(adata))

@flow(validate_parameters=False)
def mainflow(adata: sc.AnnData):
    print(adata.obs.columns)
    print(adata.var.columns)

No luck unfortunately! same thing. I'll take a look at which anndata field isn't serializing and maybe I'll report back. Thanks for the support + have a good weekend!

Zanie

01/27/2023, 9:33 PM

Alas. Thanks!

3 Views

Open in Slack

Previous Next