Brett Naul
07/07/2020, 3:18 AMResult
directly from within a task? I'm thinking about how a papermill task might look; since it runs in a subprocess you'd need to load the results there instead of in the main prefect taskrunner process....Laura Lorenz (she/her)
07/07/2020, 1:47 PMBrett Naul
07/07/2020, 4:50 PMLaura Lorenz (she/her)
07/08/2020, 2:29 PMResult
objects instead of the upstream data dependencies themselves?Brett Naul
07/08/2020, 2:52 PMexecute_notebook
. for anything larger or that isn't a built-in Python type (say a large dataframe), we've had to hack together various crazy workarounds to get those into papermill; what I really want is to pass in output=some_task()
for the parameters
cell to be populated with just output = GCSResult(bucket).read(path_to_result).value
, which means the data is only deserialized once inside the subprocess and also doesn't have to go through JSON inbetween