https://prefect.io logo
p

psimakis

10/16/2020, 3:18 PM
Hello all, Is there any way to dynamically provide the
webhook_secret
on a
SlackTask
using a
PrefectSecret
? I have tried to provide the
webhook_secret
on the instantiation but I get a
TypeError
Given this flow:
Copy code
with Flow(name='flow name') as flow:
    slack_custom_webhook_url = PrefectSecret('SLACK_E_SPIDER_STATISTICS')
    spider_name = Parameter('spider_name', default='spider a')
    with case(spider_name, 'spider a'):
        SlackTask(webhook_secret=slack_custom_webhook_url)(message='Test')
Check out the full trace-back in the thread:
n

nicholas

10/16/2020, 3:45 PM
I think I see the issue @psimakis but would you mind putting your stacktrace in this thread instead of in the main channel?
a

Alexander

10/16/2020, 4:15 PM
Copy code
SlackTask()(message='Test', webhook_secret=slack_custom_webhook_url)
Try this
p

psimakis

10/16/2020, 4:33 PM
@nicholas sure! full trace-back:
Copy code
Unexpected error: TypeError('Object of type PrefectSecret is not JSON serializable')
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/client/secrets.py", line 137, in get
    value = secrets[self.name]
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/box/box.py", line 471, in __getitem__
    raise BoxKeyError(str(err)) from None
box.exceptions.BoxKeyError: '<Task: SLACK_E_SPIDER_STATISTICS>'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 825, in get_task_run_state
    self.task.run, timeout=self.task.timeout, **raw_inputs
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/utilities/executors.py", line 188, in timeout_handler
    return fn(*args, **kwargs)
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/utilities/tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/tasks/notifications/slack_task.py", line 51, in run
    webhook_url = cast(str, Secret(self.webhook_secret).get())
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/client/secrets.py", line 147, in get
    variables=dict(name=self.name),
  File "/home/user/.local/share/virtualenvs/data-workflows-GfPV92cZ/lib/python3.7/site-packages/prefect/client/client.py", line 279, in graphql
    params=dict(query=parse_graphql(query), variables=json.dumps(variables)),
  File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PrefectSecret is not JSON serializable
@Alexander well 
SlackTask.run
 doesn't take 
webhook_secret
 as argument: 
TypeError: got an unexpected keyword argument 'webhook_secret'
n

nicholas

10/16/2020, 4:48 PM
thanks @psimakis ! I think the best option in this case would be to wrap the
SlackTask
in your own task and return its instantiation, so something like this:
Copy code
Class DynamicSlackTask(SlackTask):
  def run(self, webhook_url, message):
    self.webhook_url = webhook_url
    super(DynamicSlackTask).run(message=message)

with Flow(name='flow name') as flow:
    slack_custom_webhook_url = PrefectSecret('SLACK_E_SPIDER_STATISTICS')
    spider_name = Parameter('spider_name', default='spider a')
    with case(spider_name, 'spider a'):
        DynamicSlackTask()(webhook_secret=slack_custom_webhook_url, message='Test')
You're welcome to PR that task in the Task Library to add webhook_secret as a run parameter 😉
p

psimakis

10/16/2020, 4:50 PM
Thank you @nicholas. of course I will 🙂
🙏 1
n

nicholas

10/16/2020, 4:54 PM
Let me know if that doesn't work for you @psimakis
p

psimakis

10/16/2020, 5:10 PM
n

nicholas

10/16/2020, 5:11 PM
amazing, thanks @psimakis 😄 the core team will look at that when they get a chance 🙂
👍 1
p

psimakis

10/16/2020, 11:00 PM
@nicholas approach works fine! nil:
self
should be provided in `super`:
super(DynamicSlackTask, self).run(message=message)
. Thank you!
n

nicholas

10/16/2020, 11:00 PM
Ah oops, oversight on my part sorry!