Julius Faber
08/04/2025, 9:07 AMMarvin
08/04/2025, 9:07 AMMarvin
08/04/2025, 9:08 AMRecursionError
you're seeing suggests that FastAPI's jsonable_encoder
is getting stuck in an infinite loop while trying to serialize your decorated class. This typically happens when:
- There are circular references in your object
- The object has nested attributes that reference back to parent objects
- The decorator is creating a complex object structure that can't be easily serialized
2. Restrictions on Flow Arguments:
- Arguments must be serializable (JSON-compatible by default)
- Should not exceed 512kb in size
- Should not contain circular references
- Should be relatively simple data structures (primitives, lists, dicts, etc.)
3. Solutions:
python
# Option 1: Create the class instance outside the flow
@dataclass
class MyClass:
# your class definition
pass
decorated_instance = your_decorator(some_function)
@flow
def my_flow(instance=decorated_instance):
# Use the instance here
pass
# Option 2: Use Pydantic for serialization
from pydantic import BaseModel
class MyWrapper(BaseModel):
# Define your wrapper with proper serialization
class Config:
arbitrary_types_allowed = True # If needed
@flow
def my_flow(wrapped_instance: MyWrapper):
# Use the instance here
pass
4. Best Practices:
- Keep flow arguments simple and serializable
- Pass references or identifiers instead of complex objects when possible
- If you need to pass complex objects, consider implementing proper serialization methods
- Use Pydantic models to handle complex types
Could you share a simplified version of your decorator and how you're using it with the flow? This would help me provide more specific guidance for your use case.
Also, as a workaround, you might want to consider:
1. Creating the decorated class outside the flow and passing simpler parameters
2. Using Pydantic models to handle the serialization
3. Implementing __getstate__
and __setstate__
methods on your class if you need custom serialization behaviorJulius Faber
08/04/2025, 9:12 AMMarvin
08/04/2025, 9:12 AMMarvin
08/04/2025, 9:13 AMpython
# DON'T: Pass non-serializable objects directly
@task
def bad_task(my_complex_object): # Could cause recursion or serialization errors
...
# DO: Pass serializable data or references
@task
def good_task(object_id: str): # Pass an identifier instead
complex_object = load_object(object_id)
...
# DO: Use Pydantic for complex objects
from pydantic import BaseModel
class SerializableWrapper(BaseModel):
data: dict
class Config:
arbitrary_types_allowed = True
@task
def better_task(wrapper: SerializableWrapper):
...
4. Best Practices for Task Arguments:
- Keep arguments simple and serializable when possible
- Use references (IDs, paths) instead of complex objects
- If you need to pass complex objects:
- Implement proper serialization methods
- Use Pydantic models
- Consider breaking the task into smaller pieces
- Be especially careful with:
- File handles
- Database connections
- Custom class instances
- Decorated objects (as in your case)
5. For Your Specific Case:
If you're working with a decorator that creates a class, consider:
- Creating the decorated instance outside the task
- Passing only the necessary data into the task
- Implementing proper serialization methods if you must pass the decorated object
Remember that task arguments need to be even more carefully considered than flow arguments because they're more likely to be executed in distributed environments and their results need to be stored in the Prefect backend.