Hi all, Working in GKE k8 environment. I have this issue where my pod will go down and the main deployment will restart. The issue is that the deployment will attempt to retrieve the past task results but it is not able to causing the common issue:

21:16:34.430 | ERROR   | Flow run 'tricky-alligator' - Finished in state Failed('Flow run encountered an exception. MissingResult: The result was not persisted and is no longer available.\n')

My guess is that new pod or new start causes all persisted data to go away. Is there something I should be doing to address this? Thanks k8 gang

hey @John Horn - where are you persisting your results? if you're not using GCS for this

My guess is that new pod or new start causes all persisted data to go away.

that might be something you wanna check out - that way they can live according to lifecycle rules or something

I started persisting now to GCS to see if that will help

nice! that's what i would do too

One question I have is when persisting a sub-flow result, I noticed you can't specify the serializer

I noticed you can't specify the serializer

you should be able to, like this

In [5]: from prefect import flow

In [6]: @flow(persist_result=True)
   ...: def child():
   ...:     return "something important"
   ...:

In [7]: @flow
   ...: def parent():
   ...:     child.with_options(result_serializer=...)()
   ...:

are you using

persist_result=True

on the subflow? or i guess you dont need

with_options

here unless you wanna use different serializers in different cases, you could just do it in the decorator of

child

right away as well

If I try something like:

@flow(
    persist_result=True,
    cache_expiration=timedelta(minutes=15),
    result_storage=GCS(
        bucket_path='yo-bucket',
        project='foobar-project',           service_account_info=os.environ['gcs-creds']
    )
)
def foobar_subflow(

then I get back:

TypeError: flow() got an unexpected keyword argument 'cache_expiration'

ahh, i see.

That said haven't tried with the persist on the subflow yet to GCS

i would try that otherwise there's 2 workarounds I could think of (but I anticipate the above working) • GCS lifecycle rules • deploy the subflow, wrap

run_deployment

in a task with

cache_expiration

and call that from your parent

if I set the subflow to persist_result=False then if the pod goes down it should just re-run that specific subflow right?

well if you're calling the subflow as a python function it doesnt get its own infra (pod) it runs on the same pod as the parent

I guess in this example

deployment_flow = parent_flow():

def parent_flow:

    @flow(persist_result=False)
    def  sub_flow_1:
        task_1

        task_2

    @flow(persist_result=False)
    def  sub_flow_2:
        task_1

        task_2

yes. I believe that is would what would happen if you had

persist_result=True

on

sub_flow_2

and the parent had to retry for some reason which would not be possible if you didnt have remote result storage, since if you had

persist_result=True

but "local" result storage, it would die along with the pod

gah now I'm doubting this is true because I already has persist_result = False on sub_flow_1 and was getting that persist error

sure - feel free to bring back any roadblocks