as
09/23/2020, 12:51 PMjsonresult = task(
get_json_fun,
result=LocalResult(serializer=JSONSerializer()),
target='/path/result.json',
checkpoint=True,
)(p)
Is jsonresult still in memory next to being written to result.json or not?
I'm worried that my machine will get out of memory when my flow get's bigger with many task results floating around.josh
09/23/2020, 1:24 PMas
09/23/2020, 1:28 PMjosh
09/23/2020, 1:30 PMfinal1 = flow.run()
final2 = flow.run()
final3 = flow.run()
Then as long as that process is alive all three of those states w/ task results will exist in memory.
If you run your flow as a script like:
python my_flow_file.py
Then the results that were in memory will no longer exist in memory because the process ended.
In both cases the results will still be written to a local file using the LocalResult.as
09/23/2020, 1:45 PMjosh
09/23/2020, 1:52 PMas
09/23/2020, 2:05 PMjosh
09/23/2020, 2:15 PMas
09/23/2020, 2:21 PMjosh
09/23/2020, 2:24 PM@task
def a():
d = get_data(...)
location = write_to_file(d)
return location
@task
def b(location):
d = load_from_file(location)
...
with Flow(...) as f:
location = a()
b(location)
^ this only passes around the location of the dataas
09/23/2020, 2:33 PMjosh
09/23/2020, 2:36 PMMarvin
09/23/2020, 2:37 PMas
09/23/2020, 3:42 PM