https://prefect.io logo
#prefect-server
Title
# prefect-server
m

Michał Augoff

03/30/2022, 1:43 PM
Hi all! I see some inconsistencies in terms of task results when using S3 storage (script based) vs Docker storage and wanted to double check if I’m missing something here. My flow looks roughly like this:
Copy code
def get_flow(...):
  with Flow(...) as f:
    tasks
  return f
then I use it in another file for deployment and set some env-specific configuration:
Copy code
from module_x import get_flow

f = get_flow()
f.result = S3Result(bucket=dev or prod bucket here)

# no results with this storage configuration
f.storage = S3(bucket & key, stored_as_script=True, local_script_path=<this file>)

# results work with this storage
f.storage = Docker(...)

f.register()
With S3 storage I don’t get any results saved in S3, everything works fine when using Docker storage. I wonder if this has anything to do with Docker using serialization and S3 using the “as script” approach
a

Anna Geller

03/30/2022, 1:44 PM
can you share the storage configuration? 🙂
you asked about task results - Results are an entirely different concept than Storage in Prefect 1.0. Docker storage won't store any results, but you can use S3Result class with both S3 and Docker storage
m

Michał Augoff

03/30/2022, 1:47 PM
sorry I messed up with new lines and my message was posted too early 😅 now the example is complete
on the task level I see this with the Docker storage (1st screenshot) and the S3 storage (2nd)
you asked about task results - Results are an entirely different concept than Storage in Prefect 1.0. Docker storage won’t store any results, but you can use S3Result class with both S3 and Docker storage
this comment was probably related to my incomplete example but just to clarify - I set S3Result in both cases, but with S3 storage somehow it’s not getting picked up at registration, while it works fine if I use Docker storage
not sure if this is related in any way: https://github.com/PrefectHQ/prefect/issues/4252
k

Kevin Kho

03/30/2022, 2:10 PM
Yes that is related. This happens because
flow.register()
gets metadata and pushes it to Prefect cloud. The metadata can be found here. Notice executor and results are not part of it so we don’t store that. I think because result can contain sensitive info like the bucket. So it gets loaded in from the Flow file so you need to define it there
m

Michał Augoff

03/30/2022, 2:19 PM
ah that makes more sense now! so I assume when I use Docker storage, it works because the flow gets serialized into the storage including the result (because the entire flow Python object is serialized), but when I use script-based S3 storage, all the dynamic settings not directly defined in
get_flow
are not picked up. Would this reasoning be correct?
k

Kevin Kho

03/30/2022, 2:21 PM
Yes exactly
m

Michał Augoff

03/30/2022, 2:35 PM
👍 thanks for clarifying, that was a bit confusing, I tested S3Storage with
stored_as_script=False
and the results work now
👍 1
7 Views