Ching
09/15/2022, 7:10 PMfrom prefect import flow
import pandas as pd
import numpy as np
@flow
def do_nothing(df):
return None
if __name__ == "__main__":
df = pd.DataFrame(np.random.random((10_000, 20)))
do_nothing(df)
Traceback in threadTraceback (most recent call last):
File "/test_big_df.py", line 13, in <module>
do_nothing(df)
File "/lib/python3.9/site-packages/prefect/flows.py", line 384, in __call__
return enter_flow_run_engine_from_flow_call(
File "/lib/python3.9/site-packages/prefect/engine.py", line 158, in enter_flow_run_engine_from_flow_call
return anyio.run(begin_run)
File "/lib/python3.9/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/lib/python3.9/site-packages/prefect/client.py", line 103, in with_injected_client
return await fn(*args, **kwargs)
File "/lib/python3.9/site-packages/prefect/engine.py", line 215, in create_then_begin_flow_run
flow_run = await client.create_flow_run(
File "/lib/python3.9/site-packages/prefect/client.py", line 609, in create_flow_run
flow_run_create_json = flow_run_create.dict(json_compatible=True)
File "/lib/python3.9/site-packages/prefect/orion/utilities/schemas.py", line 257, in dict
return json.loads(self.json(*args, **kwargs))
File "/lib/python3.9/site-packages/prefect/orion/utilities/schemas.py", line 227, in json
return super().json(*args, **kwargs)
File "pydantic/main.py", line 505, in pydantic.main.BaseModel.json
File "/lib/python3.9/site-packages/prefect/orion/utilities/schemas.py", line 123, in orjson_dumps
return orjson.dumps(v, default=default).decode()
TypeError: Dict key must be str
from prefect import flow
import pandas as pd
import numpy as np
@flow
def do_nothing(values, index, columns):
return None
if __name__ == "__main__":
df = pd.DataFrame(np.random.random((10_000, 20)))
do_nothing(df.values, df.index, df.columns)
Anna Geller
09/15/2022, 7:12 PMChing
09/15/2022, 7:13 PMAnna Geller
09/15/2022, 9:05 PMErik
09/25/2022, 8:56 AMAnna Geller
09/25/2022, 10:37 AMErik
09/25/2022, 10:37 AM<http://df.to|df.to>_json()
didn’t work. I finally got df.tocsv()
to work. And then to recreate df=pd.read_csv(StringIO(csv_str))
TypeError: Dict key must be str
<http://df.to|df.to>_json()
which is weirdAnna Geller
09/25/2022, 11:03 AMErik
09/25/2022, 11:29 AM