https://prefect.io logo
Powered by
Title
j

James Gatter

01/27/2023, 6:08 PM
Hello! I'm encountering errors when passing a non-JSON-serializable object, an AnnData, into my flows or from flow to subflow. If an object isn't serializable as JSON, does that mean it can't be passed to flows/subflows cloud deployments? Anyway, any solutions or workarounds are appreciated!
__main___py.py
__main___py.py
Trace: 
ubuntu:~/Projects/$ python -m sandbox
Fatal Python error: Cannot recover from stack overflow.
Python runtime state: initialized

Current thread 0x00007f4132713740 (most recent call first):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/dtypes/generic.py", line 41 in _check
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/dtypes/generic.py", line 47 in _instancecheck
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/construction.py", line 444 in extract_array
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/arrays/_mixins.py", line 296 in __getitem__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1832 in getitem_block_index
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 4105 in _slice
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1638 in _get_slice_axis
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1602 in _getitem_axis
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1073 in __getitem__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/anndata/_core/anndata.py", line 336 in _init_as_view
  File "/home/ubuntu/.local/lib/python3.8/site-packages/anndata/_core/anndata.py", line 289 in __init__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/anndata/_core/anndata.py", line 1114 in __getitem__
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 152 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 161 in jsonable_encoder
  File "/home/ubuntu/.local/lib/python3.8/site-packages/fastapi/encoders.py", line 117 in jsonable_encoder
  ...
Aborted (core dumped)
Other info/details: • An AnnData is essentially composed of collections of pandas DataFrames, dictionaries, and a large numpy/scipy matrix (
.X
). It's stored in h5ad files which are based on the hdf5 file format (not plain text). • Using Prefect 2.7.8 • Using a local instance of Orion (--host 0.0.0.0) on AWS EC2 • I've tried not using Pydantic model as the input param and simply the AnnData itself and I get the same result
z

Zanie

01/27/2023, 7:05 PM
This looks like a bug in FastAPI’s 
jsonable_encoder
which is used to convert flow parameters into a form that can be sent to the API
If it doesn’t cause a stack overflow, we can probably capture the error and note that this parameter is unserializable
However, this is quite the crash it doesn’t look like we can capture and ignore it
j

James Gatter

01/27/2023, 7:15 PM
thanks for the quick follow up Michael! what would be your suggested course of action?
z

Zanie

01/27/2023, 7:20 PM
I’d reproduce the stack overflow with just FastAPI’s jsonable encoder and open a ticket there, I’m curious if they’d fix it.
👍 1
I believe we do capture errors here, we allow unserializable parameters to be used. If they’re going to crash the whole Python process there’s not much we can do.
We could perhaps skip certain datatypes but that’s a little tricky
j

James Gatter

01/27/2023, 8:04 PM
Yeah, 
jsonable_encoder(adata)
alone is enough to cause the error. https://github.com/tiangolo/fastapi/issues/4834 is essentially the issue I had without the stack overflow Based on the thread, if I post I don't think it'll be met with good reception 😅
Only workaround I can think of is passing filepaths instead and having each workflow read the object into memory, but that's not as ideal when working on one machine. Is there any other way I can pass a non-serializable object to flows?
z

Zanie

01/27/2023, 9:00 PM
I think you could argue that crashing the process is far worse than raising an unclear exception
💯 1
Can you try annotating your parameter with 
quote
when you pass it to the flow?
your_flow(foo=prefect.utilities.annotations.quote(your_param))
We may skip JSON serialization in that case, I’m not sure though.
What datatype is causing the problem?
j

James Gatter

01/27/2023, 9:01 PM
AnnData
or do you mean exception wise? not sure because of the stack overflow
I suppose I could try all the objects within the AnnData and find out which one as well! But yes for now I will try your suggestion! Thanks
z

Zanie

01/27/2023, 9:06 PM
I see I gotta look at AnnData 🙂
j

James Gatter

01/27/2023, 9:08 PM
Haha you should if you're interested in data science!
import prefect
from prefect import task, flow, get_run_logger
import scanpy as sc

def main():
    adata = sc.read_h5ad('demuxed.h5ad')
    mainflow(adata=prefect.utilities.annotations.quote(adata))

@flow(validate_parameters=False)
def mainflow(adata: sc.AnnData):
    print(adata.obs.columns)
    print(adata.var.columns)
No luck unfortunately! same thing. I'll take a look at which anndata field isn't serializing and maybe I'll report back. Thanks for the support + have a good weekend!
z

Zanie

01/27/2023, 9:33 PM
Alas. Thanks!
#prefect-communityJoin Slack
Powered by