https://prefect.io logo
Powered by
Title
k

Kyle Foreman (Convoy)

08/29/2019, 5:16 PM
any advice on testing Retry handlers? for something that's hard for me to make intentionally fail but I want to see how the retry handler behaves is there a convenient way to set the state to failed manually?
c

Chris White

08/29/2019, 5:18 PM
Hey Kyle --> one way I do this in my own testing, I use 
task_run_count
which is found in context. So: 
if prefect.context.get("task_run_count") == 1:
   raise ValueError("fake error")
^^ will only fail on the first run; obviously you can tweak this to fail more than one time and eventually succeed, etc.
k

Kyle Foreman (Convoy)

08/29/2019, 5:20 PM
ah nice, thanks!
šŸ‘ 1
can a retry handler not call a task method?
e.g. 
def full_refresh_on_retry(dbt_task, old_state, new_state):
    if isinstance(new_state, state.Retrying):
        dbt_task.__recompile_node(full_refresh=True)
    return new_state
c

Chris White

08/29/2019, 5:37 PM
yea that should work -> are you seeing otherwise?
k

Kyle Foreman (Convoy)

08/29/2019, 5:37 PM
gives me:
[2019-08-29 17:34:49,545] INFO - prefect.TaskRunner | Unexpected error: ValueError('Fake error to test retries',)
[2019-08-29 17:34:51,865] ERROR - prefect.TaskRunner | Exception raised while calling state handlers: AttributeError("'DbtTask' object has no attribute '__recompile_node'",)
if I call that method directly inside of my run method (not as part of a retry handler) it works
c

Chris White

08/29/2019, 5:45 PM
hmmm thatā€™s surprising; the task object itself is passed directly to state handlers
k

Kyle Foreman (Convoy)

08/29/2019, 5:48 PM
weird - and strange that it's raising an attribute error when attempting to call a method
I'll do some debugging after lunch
c

Chris White

08/29/2019, 5:52 PM
yea, something seems suspicious there -> you could drop into a debug session inside your handler with 
from IPython import embed; embed()
and make sure 
dbt_task
is what we think it is
k

Kyle Foreman (Convoy)

08/29/2019, 5:53 PM
Will do, thanks
weird, it has it but prefixed with the class name....
(Pdb++) dir(dbt_task)                                                                                         
['_DbtTask__build_runner', '_DbtTask__recompile_node', '__add__', '__and__', '__call__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__mifflin__', '__mod__', '__module__', '__mul__', '__ne__', '__new__', '__or__', '__pow__', '__radd__', '__rand__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rsub__', '__rtruediv__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__weakref__', '_compiled_node', '_dbt_flow', '_node_type', '_runner', 'auto_generated', 'bind', 'cache_for', 'cache_key', 'cache_validator', 'checkpoint', 'copy', 'inputs', 'is_equal', 'is_not_equal', 'logger', 'map', 'max_retries', 'name', 'not_', 'or_', 'outputs', 'result_handler', 'retry_delay', 'run', 'serialize', 'set_dependencies', 'set_downstream', 'set_upstream', 'skip_on_upstream_skip', 'slug', 'state_handlers', 'tags', 'timeout', 'trigger']
c

Chris White

08/29/2019, 7:42 PM
OH
k

Kyle Foreman (Convoy)

08/29/2019, 7:42 PM
'_DbtTask__recompile_node'
is the method
c

Chris White

08/29/2019, 7:42 PM
haha yes you have found a bizarre implementation detail of Python, link incomingā€¦
k

Kyle Foreman (Convoy)

08/29/2019, 7:42 PM
this a Super thing?
c

Chris White

08/29/2019, 7:43 PM
kind of; 
In [3]: class A: 
   ...:     __x = 5 
   ...:                                                                                                                                                                        

In [4]: a = A()                                                                                                                                                                

In [5]: a.__x                                                                                                                                                                  
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-99a8eeb8b502> in <module>
----> 1 a.__x

AttributeError: 'A' object has no attribute '__x'
sorry for the bad formatting
but python renames things with leading double underscores
k

Kyle Foreman (Convoy)

08/29/2019, 7:44 PM
oh right...
my bad for getting cute with private methods
thanks!
c

Chris White

08/29/2019, 7:44 PM
haha no worries --> anytime!
yea I think itā€™s called ā€œvariable manglingā€ --> if you add two trailing underscores as well, it should work fine
i think its just for things like 
__X
and 
__X_
k

Kyle Foreman (Convoy)

08/29/2019, 7:47 PM
alright, that fixed it!
c

Chris White

08/29/2019, 7:47 PM
šŸ’Æ
j

Jeremiah

08/29/2019, 11:38 PM
MANGLING STRIKES AGAINā€¦ this was something fun that @Chris White and I dealt with more than a year ago
šŸ˜‚ 3
Hardly documented, extremely annoyingā€¦ but Iā€™m sure useful to someoneā€¦ perhaps.
k

KJ

08/30/2019, 12:11 AM
Ya mangling can be problematic šŸ¤¦ā€ā™‚ļø
k

Kyle Foreman (Convoy)

08/30/2019, 1:30 AM
haha, it's one of those things I definitely learned at some point years ago and figured I would never need to think about again
#prefect-communityJoin Slack
Powered by