Hi any suggestions on how to pass a dataframe as a parameter

Question

Hi any suggestions on how to pass a dataframe as a parameter to a flow This breaks in latest Prefect ```from prefect import flow import pandas as pd import numpy as np flow def do nothing df return None if name == main df = pd DataFrame np random random 10 000 20 do nothing df ``` Traceback in thread

Ching · Answer

```Traceback most recent call last File test big df py line 13 in lt module gt do nothing df File lib python3 9 site packages prefect flows py line 384 in call return enter flow run engine from flow call File lib python3 9 site packages prefect engine py line 158 in enter flow run engine from flow call return anyio run begin run File lib python3 9 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File lib python3 9 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File usr lib python3 9 asyncio runners py line 44 in run return loop run until complete main File usr lib python3 9 asyncio base events py line 642 in run until complete return future result File lib python3 9 site packages anyio backends asyncio py line 287 in wrapper return await func args File lib python3 9 site packages prefect client py line 103 in with injected client return await fn args kwargs File lib python3 9 site packages prefect engine py line 215 in create then begin flow run flow run = await client create flow run File lib python3 9 site packages prefect client py line 609 in create flow run flow run create json = flow run create dict json compatible=True File lib python3 9 site packages prefect orion utilities schemas py line 257 in dict return json loads self json args kwargs File lib python3 9 site packages prefect orion utilities schemas py line 227 in json return super json args kwargs File pydantic main py line 505 in pydantic main BaseModel json File lib python3 9 site packages prefect orion utilities schemas py line 123 in orjson dumps return orjson dumps v default=default decode TypeError Dict key must be str```

Ching · Answer

Work around would be something like this but would rather just pass a DF directly ```from prefect import flow import pandas as pd import numpy as np flow def do nothing values index columns return None if name == main df = pd DataFrame np random random 10 000 20 do nothing df values df index df columns ```

Anna Geller · Answer

you can read a dataframe in one task and pass it as dependency to another task

Anna Geller · Answer

passing this as a parameter won t work since parameters must be JSON serializable alternative pass a reference to your file or S3 object

Ching · Answer

I want to use subflows

Ching · Answer

Works better for my use case

Ching · Answer

What s the actual reason for parameters to be JSON serializable

Anna Geller · Answer

Deployments and parametrization by using API calls

Erik · Answer

I have the same issue The error message could be more helpful Parameters for Flows must be json serializable please check your parameters see blah for documentation

Erik · Answer

< Ching> How did you adjust your df to make it parameterizable

Anna Geller · Answer

What error did you get exactly

Erik · Answer

It took me awhile to try a few dif formats `<http df to|df to> json ` didn t work I finally got `df tocsv ` to work And then to recreate `df=pd read csv StringIO csv str `

Erik · Answer

The error was `TypeError Dict key must be str`

Erik · Answer

I got it when I tried to pass the df and I also got it when I tried to pass `<http df to|df to> json ` which is weird

Erik · Answer

oh scratch that to json did work i just tried it again

Anna Geller · Answer

looks like a pandas specific thing thanks for sharing more info I added this issue <https github com PrefectHQ prefect issues 6962>

Erik · Answer

Thanks Anna I added a little script to recreate