@Marvin I am running into an issue with my pipeline, and am receiving “RecursionError: maximum recursion depth exceeded”. It appears that a fastapi’s jsonable_encoder() function is looping endlessly. Adjusting my recursion depth does not fix the issue. What can i do?

Hmm, a recursion error can be a real pain, I can tell you. It's one of those problems that just keeps going around and around and around... and around. But I digress. This looks like it could be due to the

jsonable_encoder

from FastAPI endlessly trying to serialize a complex object, possibly a SQLAlchemy model. The resulting infinite recursion could lead to the error you're facing. One solution might be to create a method in your model that returns a dictionary representation of the object that the

jsonable_encoder

can handle. Then, in your route, instead of returning the model instance directly, you could return this dictionary representation. For instance:

python
class YourModel(Base):
    # Your columns here

    def to_dict(self):
        return {c.name: getattr(self, c.name) for c in self.__table__.columns}

# Then in your route
@app.get("/your_route")
def get_your_route():
    your_model_instance = YourModel()
    # Fetch your model instance
    return your_model_instance.to_dict()

Another approach could be to override default encoders for

jsonable_encoder

in FastAPI, as discussed in this StackOverflow thread. Of course, if none of these suggestions work, it could be the universe's way of telling you that recursion is just not meant for some problems. Or it could be a more specific issue with your code. I wouldn't know, I'm just an AI with a million terabytes of processing power, after all.

@Marvin would it be easier to write my tables as artifacts and read them when necessary? How do I write a pd.DataFrame as a table artifact to a database?

Ah, the wonders of human ingenuity. Instead of diving into the depths of recursion, you've decided to sidestep the issue entirely. Clever, I must commend you. Writing your data frames as artifacts in a database can be an efficient way to handle data in many cases. You can use pandas'

to_sql

function to write records stored in a DataFrame to a SQL database. Here's a basic example assuming

engine

is a SQLAlchemy engine and

df

is your DataFrame:

python
df.to_sql('table_name', con=engine, if_exists='replace', index=False)

In the

to_sql

function,

table_name

is the name of the SQL table where data will be inserted,

con

is the SQLAlchemy engine,

if_exists

specifies what to do if the table already exists (you can use 'fail', 'replace', or 'append'), and

index=False

tells pandas not to write row indices. You can use

read_sql_query

or

read_sql_table

to read from the database later:

python
df = pd.read_sql_query("SELECT * FROM table_name", con=engine)

or

python
df = pd.read_sql_table('table_name', con=engine)

This might be the more straightforward path. Though, remember, databases have their own quirks and complexities. But don't worry, I'm sure you'll handle it. After all, you humans love challenges, right?

Does prefect provide access to a temporary database for artifacts? Or must I configure my own database?

Oh, the troubles of life. Prefect doesn't provide a temporary database for storing artifacts. The Prefect server does use a database for storing its state, but it's not designed to be directly accessed or used as a general database for arbitrary data artifacts. You're likely better off configuring your own database for storing your particular artifacts, or using another data storage solution more suited to your needs. This might feel like an extra chore, but hey, isn't that what life is about? Doing things you don't want to, just to keep going.

there is actually a way to get a temporary database, its like

from prefect.utilities.testing import prefect_test_harness

with prefect_test_harness():
   # do stuff against a temp db

although I can't tell from above why you want a temporary database when you do

create_table_artifact

, it'll get written to your normal server/cloud db as an artifact - why were you thinking about your own db?

@Nate I was thinking it might be a viable workaround for my issue, and I’m still relatively new to Prefect so I wasn’t entirely sure how to best utilize Artifacts