^^^^^^^^^^^^^^^^^ File "/home/dju/code/dlthub/dl...
# ask-marvin
j
^^^^^^^^^^^^^^^^^ File "/home/dju/code/dlthub/dlt-plus/.venv/lib/python3.12/site-packages/fastapi/encoders.py", line 333, in jsonable_encoder return jsonable_encoder( ^^^^^^^^^^^^^^^^^ File "/home/dju/code/dlthub/dlt-plus/.venv/lib/python3.12/site-packages/fastapi/encoders.py", line 289, in jsonable_encoder encoded_value = jsonable_encoder( ^^^^^^^^^^^^^^^^^@Marvin are there any restrictions on what kind of arguments i can pass to a flow? I have a decorator that creates a class from a function, i pass it to a flow i get max recursion depth exception:
m
thought for 56.2 seconds
Based on the research and the error you're encountering, here's what's happening and how to address it: 1. The Root Issue: The
RecursionError
you're seeing suggests that FastAPI's
jsonable_encoder
is getting stuck in an infinite loop while trying to serialize your decorated class. This typically happens when: - There are circular references in your object - The object has nested attributes that reference back to parent objects - The decorator is creating a complex object structure that can't be easily serialized 2. Restrictions on Flow Arguments: - Arguments must be serializable (JSON-compatible by default) - Should not exceed 512kb in size - Should not contain circular references - Should be relatively simple data structures (primitives, lists, dicts, etc.) 3. Solutions:
Copy code
python
# Option 1: Create the class instance outside the flow
@dataclass
class MyClass:
    # your class definition
    pass

decorated_instance = your_decorator(some_function)

@flow
def my_flow(instance=decorated_instance):
    # Use the instance here
    pass

# Option 2: Use Pydantic for serialization
from pydantic import BaseModel

class MyWrapper(BaseModel):
    # Define your wrapper with proper serialization
    class Config:
        arbitrary_types_allowed = True  # If needed

@flow
def my_flow(wrapped_instance: MyWrapper):
    # Use the instance here
    pass
4. Best Practices: - Keep flow arguments simple and serializable - Pass references or identifiers instead of complex objects when possible - If you need to pass complex objects, consider implementing proper serialization methods - Use Pydantic models to handle complex types Could you share a simplified version of your decorator and how you're using it with the flow? This would help me provide more specific guidance for your use case. Also, as a workaround, you might want to consider: 1. Creating the decorated class outside the flow and passing simpler parameters 2. Using Pydantic models to handle the serialization 3. Implementing
__getstate__
and
__setstate__
methods on your class if you need custom serialization behavior
j
@Marvin do the same recommendations apply to task args?
m
thought for 49.0 seconds
Yes, the same serialization requirements apply to task arguments, but with some important nuances. Here's what you need to know: 1. Core Requirements (Same as Flows): - Arguments must be serializable - Should not contain circular references - Should be relatively simple data structures where possible 2. Key Differences for Tasks: - Tasks are more likely to be executed in distributed environments, making serialization even more critical - Tasks can receive results from other tasks, which Prefect handles automatically - Task results are stored in the backend, requiring reliable serialization/deserialization 3. Common Issues and Solutions:
Copy code
python
# DON'T: Pass non-serializable objects directly
@task
def bad_task(my_complex_object):  # Could cause recursion or serialization errors
    ...

# DO: Pass serializable data or references
@task
def good_task(object_id: str):  # Pass an identifier instead
    complex_object = load_object(object_id)
    ...

# DO: Use Pydantic for complex objects
from pydantic import BaseModel

class SerializableWrapper(BaseModel):
    data: dict
    class Config:
        arbitrary_types_allowed = True

@task
def better_task(wrapper: SerializableWrapper):
    ...
4. Best Practices for Task Arguments: - Keep arguments simple and serializable when possible - Use references (IDs, paths) instead of complex objects - If you need to pass complex objects: - Implement proper serialization methods - Use Pydantic models - Consider breaking the task into smaller pieces - Be especially careful with: - File handles - Database connections - Custom class instances - Decorated objects (as in your case) 5. For Your Specific Case: If you're working with a decorator that creates a class, consider: - Creating the decorated instance outside the task - Passing only the necessary data into the task - Implementing proper serialization methods if you must pass the decorated object Remember that task arguments need to be even more carefully considered than flow arguments because they're more likely to be executed in distributed environments and their results need to be stored in the Prefect backend.