John Horn
07/05/2023, 9:55 PM21:16:34.430 | ERROR | Flow run 'tricky-alligator' - Finished in state Failed('Flow run encountered an exception. MissingResult: The result was not persisted and is no longer available.\n')
My guess is that new pod or new start causes all persisted data to go away.
Is there something I should be doing to address this? Thanks k8 gangNate
07/06/2023, 1:00 AMMy guess is that new pod or new start causes all persisted data to go away.that might be something you wanna check out - that way they can live according to lifecycle rules or something
John Horn
07/06/2023, 1:01 AMNate
07/06/2023, 1:01 AMJohn Horn
07/06/2023, 1:01 AMNate
07/06/2023, 1:07 AMI noticed you can't specify the serializeryou should be able to, like this
In [5]: from prefect import flow
In [6]: @flow(persist_result=True)
...: def child():
...: return "something important"
...:
In [7]: @flow
...: def parent():
...: child.with_options(result_serializer=...)()
...:
are you using persist_result=True
on the subflow?
or i guess you dont need with_options
here unless you wanna use different serializers in different cases, you could just do it in the decorator of child
right away as wellJohn Horn
07/06/2023, 1:10 AM@flow(
persist_result=True,
cache_expiration=timedelta(minutes=15),
result_storage=GCS(
bucket_path='yo-bucket',
project='foobar-project', service_account_info=os.environ['gcs-creds']
)
)
def foobar_subflow(
then I get back:
TypeError: flow() got an unexpected keyword argument 'cache_expiration'
prefect.exceptions.MissingResult: The result was not persisted and is no longer availabl
and the stack trace leads up to the subflow:
blah_output = foobar_subflow(
Nate
07/06/2023, 1:15 AMThat said haven't tried with the persist on the subflow yet to GCSi would try that otherwise there's 2 workarounds I could think of (but I anticipate the above working) ⢠GCS lifecycle rules ⢠deploy the subflow, wrap
run_deployment
in a task with cache_expiration
and call that from your parentJohn Horn
07/06/2023, 1:17 AMNate
07/06/2023, 1:17 AMrun_deployment
, then that flow run gets its own infraJohn Horn
07/06/2023, 1:19 AMdeployment_flow = parent_flow():
def parent_flow:
@flow(persist_result=False)
def sub_flow_1:
task_1
task_2
@flow(persist_result=False)
def sub_flow_2:
task_1
task_2
Nate
07/06/2023, 1:23 AMpersist_result=True
on sub_flow_2
and the parent had to retry for some reason
which would not be possible if you didnt have remote result storage, since if you had persist_result=True
but "local" result storage, it would die along with the podJohn Horn
07/06/2023, 1:24 AMNate
07/06/2023, 1:25 AM