Channels
#prefect-community
Title
k
Kyle Foreman (Convoy)
08/29/2019, 5:16 PMany advice on testing Retry handlers? for something that's hard for me to make intentionally fail but I want to see how the retry handler behaves is there a convenient way to set the state to failed manually?
c
Chris White
08/29/2019, 5:18 PM
Hey Kyle --> one way I do this in my own testing, I use
task_run_count Copy code
if prefect.context.get("task_run_count") == 1:
raise ValueError("fake error")k
Kyle Foreman (Convoy)
08/29/2019, 5:20 PMah nice, thanks!
š 1
can a retry handler not call a task method?
e.g.
Copy code
def full_refresh_on_retry(dbt_task, old_state, new_state):
if isinstance(new_state, state.Retrying):
dbt_task.__recompile_node(full_refresh=True)
return new_statec
Chris White
08/29/2019, 5:37 PM
yea that should work -> are you seeing otherwise?
k
Kyle Foreman (Convoy)
08/29/2019, 5:37 PMgives me:
Copy code
[2019-08-29 17:34:49,545] INFO - prefect.TaskRunner | Unexpected error: ValueError('Fake error to test retries',)
[2019-08-29 17:34:51,865] ERROR - prefect.TaskRunner | Exception raised while calling state handlers: AttributeError("'DbtTask' object has no attribute '__recompile_node'",)if I call that method directly inside of my run method (not as part of a retry handler) it works
c
Chris White
08/29/2019, 5:45 PM
hmmm thatās surprising; the task object itself is passed directly to state handlers
k
Kyle Foreman (Convoy)
08/29/2019, 5:48 PMweird - and strange that it's raising an attribute error when attempting to call a method
I'll do some debugging after lunch
c
Chris White
08/29/2019, 5:52 PM
yea, something seems suspicious there -> you could drop into a debug session inside your handler with
from IPython import embed; embed()dbt_taskk
Kyle Foreman (Convoy)
08/29/2019, 5:53 PMWill do, thanks
weird, it has it but prefixed with the class name....
Copy code
(Pdb++) dir(dbt_task)
['_DbtTask__build_runner', '_DbtTask__recompile_node', '__add__', '__and__', '__call__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__mifflin__', '__mod__', '__module__', '__mul__', '__ne__', '__new__', '__or__', '__pow__', '__radd__', '__rand__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rsub__', '__rtruediv__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__weakref__', '_compiled_node', '_dbt_flow', '_node_type', '_runner', 'auto_generated', 'bind', 'cache_for', 'cache_key', 'cache_validator', 'checkpoint', 'copy', 'inputs', 'is_equal', 'is_not_equal', 'logger', 'map', 'max_retries', 'name', 'not_', 'or_', 'outputs', 'result_handler', 'retry_delay', 'run', 'serialize', 'set_dependencies', 'set_downstream', 'set_upstream', 'skip_on_upstream_skip', 'slug', 'state_handlers', 'tags', 'timeout', 'trigger']c
Chris White
08/29/2019, 7:42 PM
OH
k
Kyle Foreman (Convoy)
08/29/2019, 7:42 PM'_DbtTask__recompile_node'c
Chris White
08/29/2019, 7:42 PM
haha yes you have found a bizarre implementation detail of Python, link incomingā¦
k
Kyle Foreman (Convoy)
08/29/2019, 7:42 PMthis a Super thing?
c
Chris White
08/29/2019, 7:43 PM
kind of;
Copy code
In [3]: class A:
...: __x = 5
...:
In [4]: a = A()
In [5]: a.__x
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-5-99a8eeb8b502> in <module>
----> 1 a.__x
AttributeError: 'A' object has no attribute '__x'sorry for the bad formatting
but python renames things with leading double underscores
k
Kyle Foreman (Convoy)
08/29/2019, 7:44 PMoh right...
my bad for getting cute with private methods
thanks!
c
Chris White
08/29/2019, 7:44 PM
haha no worries --> anytime!
yea I think itās called āvariable manglingā --> if you add two trailing underscores as well, it should work fine
i think its just for things like
__X__X_k
Kyle Foreman (Convoy)
08/29/2019, 7:47 PMalright, that fixed it!
c
Chris White
08/29/2019, 7:47 PM
šÆ
j
Jeremiah
08/29/2019, 11:38 PM
MANGLING STRIKES AGAINā¦ this was something fun that @Chris White and I dealt with more than a year ago
š 3
Hardly documented, extremely annoyingā¦ but Iām sure useful to someoneā¦ perhaps.
k
KJ
08/30/2019, 12:11 AMYa mangling can be problematic š¤¦āāļø
k
Kyle Foreman (Convoy)
08/30/2019, 1:30 AMhaha, it's one of those things I definitely learned at some point years ago and figured I would never need to think about again