Brett Naul07/07/2020, 3:18 AM
directly from within a task? I'm thinking about how a papermill task might look; since it runs in a subprocess you'd need to load the results there instead of in the main prefect taskrunner process....
Laura Lorenz (she/her)07/07/2020, 1:47 PM
Brett Naul07/07/2020, 4:50 PM
Laura Lorenz (she/her)07/08/2020, 2:29 PM
objects instead of the upstream data dependencies themselves?
Brett Naul07/08/2020, 2:52 PM
. for anything larger or that isn't a built-in Python type (say a large dataframe), we've had to hack together various crazy workarounds to get those into papermill; what I really want is to pass in
cell to be populated with just
, which means the data is only deserialized once inside the subprocess and also doesn't have to go through JSON inbetween
output = GCSResult(bucket).read(path_to_result).value