Thread
#prefect-community
    John Shearer

    John Shearer

    1 year ago
    Hi there! I have some (not all) tasks that need need PandasSerializer, and also need to use different result types depending on if dev or production. I have a solution for this (below), but wondered if anyone had a cleaner solution?
    condition = True
    resultType = LocalResult if condition else S3Result
    
    @task()
    def create_list():
        return [1,2,3]
    
    @task(result=resultType(serializer=PandasSerializer(file_type='parquet')))
    def to_df(a_list):
        return pd.DataFrame({'col1': a_list})
    
    with Flow(
            name="my-flow",
            result=resultType(serializer=PickleSerializer())
    ) as my_flow:
        my_list = create_list()
        my_df = to_df(my_list)
    d

    Dustin Ngo

    1 year ago
    Hi John If you need to support different support types across environments, this pattern is appropriate! I can't offer too much more in the way of suggestions here other than maybe you want to select your task options using some sort of environment variable that's easy to pick up. One way is:
    environment = 'DEV'
    task_config = {
        'DEV': LocalResult(serializer=PandasSerializer(file_type='parquet')),
    	'PROD': S3Result(serializer=PandasSerializer(file_type='parquet')),
    } 
      
    @task()  
    def create_list():  
        return [1,2,3]  
      
    @task(result=task_config[environment])  
    def to_df(a_list):  
        return pd.DataFrame({'col1': a_list})  
      
    with Flow(  
            name="my-flow",  
            result=resultType(serializer=PickleSerializer())  
    ) as my_flow:  
        my_list = create_list()  
        my_df = to_df(my_list)
    Of course, you risk issues arising in your flows that can happen outside of Prefect by handling multiple result types, so if it's possible in the future to unify them and put as much of the logic within flows and tasks as possible, you might have a better experience!
    John Shearer

    John Shearer

    1 year ago
    Thanks. That is more clear that way