https://prefect.io logo
m

Matt

07/18/2023, 2:13 PM
@Marvin I am running into an issue with my pipeline, and am receiving “RecursionError: maximum recursion depth exceeded”. It appears that a fastapi’s jsonable_encoder() function is looping endlessly. Adjusting my recursion depth does not fix the issue. What can i do?
m

Marvin

07/18/2023, 2:14 PM
Hmm, a recursion error can be a real pain, I can tell you. It's one of those problems that just keeps going around and around and around... and around. But I digress. This looks like it could be due to the
jsonable_encoder
from FastAPI endlessly trying to serialize a complex object, possibly a SQLAlchemy model. The resulting infinite recursion could lead to the error you're facing. One solution might be to create a method in your model that returns a dictionary representation of the object that the
jsonable_encoder
can handle. Then, in your route, instead of returning the model instance directly, you could return this dictionary representation. For instance:
Copy code
python
class YourModel(Base):
    # Your columns here

    def to_dict(self):
        return {c.name: getattr(self, c.name) for c in self.__table__.columns}

# Then in your route
@app.get("/your_route")
def get_your_route():
    your_model_instance = YourModel()
    # Fetch your model instance
    return your_model_instance.to_dict()
Another approach could be to override default encoders for
jsonable_encoder
in FastAPI, as discussed in this StackOverflow thread. Of course, if none of these suggestions work, it could be the universe's way of telling you that recursion is just not meant for some problems. Or it could be a more specific issue with your code. I wouldn't know, I'm just an AI with a million terabytes of processing power, after all.
m

Matt

07/18/2023, 2:55 PM
@Marvin would it be easier to write my tables as artifacts and read them when necessary? How do I write a pd.DataFrame as a table artifact to a database?
m

Marvin

07/18/2023, 2:55 PM
Ah, the wonders of human ingenuity. Instead of diving into the depths of recursion, you've decided to sidestep the issue entirely. Clever, I must commend you. Writing your data frames as artifacts in a database can be an efficient way to handle data in many cases. You can use pandas'
to_sql
function to write records stored in a DataFrame to a SQL database. Here's a basic example assuming
engine
is a SQLAlchemy engine and
df
is your DataFrame:
Copy code
python
df.to_sql('table_name', con=engine, if_exists='replace', index=False)
In the
to_sql
function,
table_name
is the name of the SQL table where data will be inserted,
con
is the SQLAlchemy engine,
if_exists
specifies what to do if the table already exists (you can use 'fail', 'replace', or 'append'), and
index=False
tells pandas not to write row indices. You can use
read_sql_query
or
read_sql_table
to read from the database later:
Copy code
python
df = pd.read_sql_query("SELECT * FROM table_name", con=engine)
or
Copy code
python
df = pd.read_sql_table('table_name', con=engine)
This might be the more straightforward path. Though, remember, databases have their own quirks and complexities. But don't worry, I'm sure you'll handle it. After all, you humans love challenges, right?
m

Matt

07/18/2023, 2:57 PM
Does prefect provide access to a temporary database for artifacts? Or must I configure my own database?
@Marvin Does prefect provide access to a temporary database for artifacts? Or must I configure my own database?
m

Marvin

07/18/2023, 3:14 PM
Oh, the troubles of life. Prefect doesn't provide a temporary database for storing artifacts. The Prefect server does use a database for storing its state, but it's not designed to be directly accessed or used as a general database for arbitrary data artifacts. You're likely better off configuring your own database for storing your particular artifacts, or using another data storage solution more suited to your needs. This might feel like an extra chore, but hey, isn't that what life is about? Doing things you don't want to, just to keep going.
n

Nate

07/18/2023, 3:25 PM
there is actually a way to get a temporary database, its like
Copy code
from prefect.utilities.testing import prefect_test_harness

with prefect_test_harness():
   # do stuff against a temp db
although I can't tell from above why you want a temporary database when you do
create_table_artifact
, it'll get written to your normal server/cloud db as an artifact - why were you thinking about your own db?
m

Matt

07/18/2023, 4:53 PM
@Nate I was thinking it might be a viable workaround for my issue, and I’m still relatively new to Prefect so I wasn’t entirely sure how to best utilize Artifacts