afausse
03/22/2020, 5:54 PMVenkat
03/23/2020, 6:19 AMpandas.DataFrame
). For every (second and subsequent) run of the flow, I need to retrieve the previous dataFrame, in order to do a merge/diff operation with the one generated in this run. Is this possible? I read up all the docs on States and ResultHandlers, but I dont see anything that persists across flow runs.Venkat
03/23/2020, 6:19 AMbardovv
03/23/2020, 7:07 AMbardovv
03/23/2020, 7:34 AMbardovv
03/23/2020, 7:56 AMArlo Bryer
03/23/2020, 10:12 AMCab Maddux
03/23/2020, 2:59 PMA
containing task X
which has input arg1
, takes a long time to run and returns a string. It uses GCSResultHandler
, with cache_for=datatime.timedelta(hours=1)
and cache_validator=prefect.engine.cache_validators.all_inputs
:
1. A run of flow A
is triggered and the input for task X
is arg1='abc'
and the output is applebananacarrot
2. 5 minutes another run of flow A
is triggered and the input for task X
is arg1='def'
and the output is danceearflight
3. 5 minutes later another run of flow A
is triggered and the input for task X
is arg1=abc
After #1 we'll have a cached value for task X
and input abc
pointing to GCS URI storing pickled applebananacarrot
. After #2 we'll have and additional cached value for task X
and input def
pointing to GCS URI storing pickled danceearflight
. Then #3 will pull GCS URI for pickled applebananacarrot
from cache. Is that correct?Amit Singh
03/23/2020, 3:27 PMflow = flow or prefect.context.get("flow", None)
if not flow:
> raise ValueError("Could not infer an active Flow context.")
E ValueError: Could not infer an active Flow context.
John Ramirez
03/23/2020, 3:27 PM0.9.8
and saw this warning
UserWarning: DEPRECATED: the result_handler Task option will be deprecated in 0.11.0, and removed in 0.12.0, in favor of the `result` option instead.
is there a time line on these versionsJohn Ramirez
03/23/2020, 4:17 PMUnexpected error: AttributeError("'S3ResultHandler' object has no attribute '_client'")
any thoughts?Matt Juszczak
03/23/2020, 4:32 PMBrad
03/23/2020, 9:32 PMinstall
with the DockerAgent
? Or, what is the easiest way to run the agent in the background ?Mark Williams
03/23/2020, 11:00 PMAdam Roderick
03/23/2020, 11:12 PM\prefect\core\flow.py:1331: UserWarning: flow.deploy() will be deprecated in an upcoming release. Please use flow.register()
UserWarning,
[2020-03-23 22:36:15,573] INFO - prefect.Docker | Building the flow's Docker storage...
On a second development machine, we get an error with not much context:
/prefect/core/flow.py:1331: UserWarning: flow.deploy() will be deprecated in an upcoming release. Please use flow.register()
UserWarning,
Expecting value: line 1 column 1 (char 0)
Aiden Price
03/24/2020, 2:30 AM@task(skip_on_upstream_skip=True)
def concat(histories: List[pd.Dataframe]):
return pd.concat(histories)
But when I call it like concatted = concat([task1, task2])
(note the manually built list from two branches of execution) then nothing happens, the concat()
function is just not called.
So trying to be clever I tried this instead;
@task(skip_on_upstream_skip=True)
def concat(*args: pd.Dataframe):
return pd.concat(args)
But it seems I’m fighting the framework there because I get this error when I try `concatted = concat(task1, task2)`;
ValueError: Tasks with variable positional arguments (*args) are not supported, because all Prefect arguments are stored as keywords. As a workaround, consider modifying the run() method to accept **kwargs and feeding the values to *args.
So is it best to go with the advice in the warning message or try a different way?
Say something like;
@task(skip_on_upstream_skip=True)
def concat(**kwargs: pd.Dataframe):
return pd.concat(x for x in kwargs.values())
Then call with concatted = concat({"one": task1, "two": task2})
bardovv
03/24/2020, 10:48 AMbardovv
03/24/2020, 1:05 PMbardovv
03/24/2020, 1:29 PMKyle Foreman (Convoy)
03/25/2020, 1:19 AMViktor Svekolkin
03/25/2020, 9:21 AMbardovv
03/25/2020, 10:21 AMprefect.context.get What is this used for?
Arlo Bryer
03/25/2020, 12:02 PMAdam Roderick
03/25/2020, 2:10 PMPreston Marshall
03/25/2020, 2:40 PMEric Hauser
03/25/2020, 5:08 PMitay livni
03/26/2020, 4:06 AMBob Colner
03/26/2020, 5:20 AMLuke Orland
03/26/2020, 3:00 PMGeorge Coyne
03/26/2020, 4:12 PM