https://prefect.io logo
Powered by
a

Andrea Nerla

02/04/2022, 3:08 PM
Hi folks, I'm receiving this error everytime I try to run a Prefect script. For context I'm trying to receive Slack's status notifications so I added the SLACK_WEBHOOK_URL in the config.toml and I've also opened prefect's init.py out of curiosity. I don't think I changed anything in it but I might have miscliccked something. 
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\box\box.py", line 516, in __getattr__
value = object.__getattribute__(self, item)
AttributeError: 'Config' object has no attribute 'datefmt'
Entire log: 
Traceback (most recent call last):
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\box\box.py", line 488, in __getitem__
return super().__getitem__(item)
KeyError: 'datefmt'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\box\box.py", line 514, in __getattr__
value = self.__getitem__(item, _ignore_default=True)
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\box\box.py", line 509, in __getitem__
raise BoxKeyError(str(err)) from _exception_cause(err)
box.exceptions.BoxKeyError: "'datefmt'"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\box\box.py", line 516, in __getattr__
value = object.__getattribute__(self, item)
AttributeError: 'Config' object has no attribute 'datefmt'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\Users\andrea.nerla\Desktop\sidal\sidal prefect\prefect test var risorsas.py", line 1, in <module>
import prefect
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\__init__.py", line 1, in <module>
import prefect.utilities
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\__init__.py", line 1, in <module>
import prefect.utilities.logging
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\logging.py", line 261, in <module>
context.logger = prefect_logger = configure_logging()
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\__init__.py", line 1, in <module>
import prefect.utilities
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\__init__.py", line 1, in <module>
import prefect.utilities.logging
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\logging.py", line 261, in <module>
context.logger = prefect_logger = configure_logging()
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\logging.py", line 258, in configure_logging
return _create_logger(name)
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\prefect\utilities\logging.py", line 230, in _create_logger
context.config.logging.format, context.config.logging.datefmt
File "C:\Users\andrea.nerla\AppData\Local\Programs\Python\Python39\lib\site-packages\box\box.py", line 530, in __getattr__
raise BoxKeyError(str(err)) from _exception_cause(err)
box.exceptions.BoxKeyError: "'Config' object has no attribute 'datefmt'"
k

Kevin Kho

02/04/2022, 3:28 PM
Are you using 
flow.run()
or is this with an agent?
a

Andrea Nerla

02/04/2022, 3:31 PM
Right now I restarted the system and I'm trying to run 
prefect agent local start
, which gives me this error. But before of it I was having the same error with a 
flow.run()
k

Kevin Kho

02/04/2022, 3:34 PM
Can you do 
pip show python-box
in your terminal and show me the output?
a

Andrea Nerla

02/04/2022, 3:35 PM
k

Kevin Kho

02/04/2022, 3:36 PM
And for 
pip show prefect
?
a

Andrea Nerla

02/04/2022, 3:37 PM
k

Kevin Kho

02/04/2022, 3:37 PM
I am on the same but can’t replicate. I would honestly just try re-installing prefect
✅ 1
a

Andrea Nerla

02/04/2022, 3:57 PM
It did resolve the error, thanks.
👍 2
@Kevin Kho for instance I discovered the error: I was adding the SLACK_WEBHOOK_URL in the wrong section of the config.toml file (I can't recall exactly which one)
k

Kevin Kho

02/07/2022, 2:51 PM
Ah I see. Thanks for circling back @Andrea Nerla
k

Kevin Otte

03/16/2022, 12:48 AM
@Andrea Nerla can you check your config and see where you've added it? Im running into the same issue too...Once we know where it should be, can we update the prefect docs?
hmm actually I ended up adding to the end of the file and its sort of working... @Kevin Kho do I need a cloud API key for this to run?
k

Kevin Kho

03/16/2022, 12:52 AM
For 
flow.run()
no. For running with an agent against Prefect Cloud yes you have to be authenticated
k

Kevin Otte

03/16/2022, 12:54 AM
hmm Im just using the example and I get this error
Untitled.txt
Untitled.txt
added
Copy code
[context.secrets]
SLACK_WEBHOOK_URL = "<https://hooks.slack.com/services/xxx>"
to config.toml and using this sample code
Copy code
from prefect import Flow, task
from prefect.utilities.notifications import slack_notifier


@task(name="1/x task", state_handlers=[slack_notifier])
def div(x):
    return 1 / x


@task(name="Add 1 task", state_handlers=[slack_notifier])
def add(x):
    return x + 1


with Flow('Add Divide Flow') as f:
    res = div(x=add(-1))


final_state = f.run()
k

Kevin Kho

03/16/2022, 12:59 AM
Your error looks so weird try creating an env variable 
export PREFECT___CLOUD____USE_LOCAL_SECRETS=true_
or in the config.toml
Copy code
[context.secrets]
SLACK_WEBHOOK_URL = "<https://hooks.slack.com/services/T015STTHK0A/B0342AESY6L/>

[cloud]
use_local_secrets=true
k

Kevin Otte

03/16/2022, 1:02 AM
still getting the same error 😞
k

Kevin Kho

03/16/2022, 1:02 AM
Trying to replicate one sec
I think your config.toml needs an empty first line. Can you try adding that?
k

Kevin Otte

03/16/2022, 1:04 AM
hmm that didnt do the trick
Untitled.txt
Untitled.txt
full config
k

Kevin Kho

03/16/2022, 1:05 AM
No though it’s working fine for me. I think there might be something wrong with your config.toml encoding or something. Let’s try this
Can you try placing this file in the 
.prefect
folder and then editing it directly to put your Slack secret?
config.toml
Ah I see you’re on server. Let me try with your config.toml one sec
I can’t replicate with your config.toml when I paste it in my file. My errors are pretty normal as expected
Something unrelated is weird though, are you on server or cloud?
Care we try a more bare config first to isolate the problem? Mine is simply:
Copy code
[context.secrets]
SLACK_WEBHOOK_URL = "<https://hooks.slack.com/services/xxx/xxx/xxx>"

[cloud]
use_local_secrets = true
And then your script works with 
flow.run()
, we don’t need a backend yet to test this
k

Kevin Otte

03/16/2022, 1:29 AM
Not sure but assuming server bc I didnt spin up a cloud
when I remove everything but those few lines
I get
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/toolz/functoolz.py", line 306, in __call__
    return self._partial(*args, **kwargs)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/notifications/notifications.py", line 316, in slack_notifier
    form_data = slack_message_formatter(tracked_obj, new_state, backend_info)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/notifications/notifications.py", line 161, in slack_message_formatter
    url = prefect.client.Client().get_cloud_url(
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/client.py", line 1008, in get_cloud_url
    tenant_slug = self.get_default_tenant_slug()
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/client.py", line 1030, in get_default_tenant_slug
    res = self.graphql({"query": {"tenant": {"id", "slug"}}})
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/client.py", line 452, in graphql
    result = <http://self.post|self.post>(
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/client.py", line 407, in post
    response = self._request(
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/client.py", line 649, in _request
    raise AuthorizationError(
prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.
[2022-03-15 21:04:02-0400] INFO - prefect.TaskRunner | Task '1/x task': Finished task run for task with final state: 'Failed'
[2022-03-15 21:04:02-0400] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
(venv39) ➜  airflow-pipelines git:(minor_refactor) ✗ python test_slack.py
Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/box/box.py", line 488, in __getitem__
    return super().__getitem__(item)
KeyError: 'logging'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/box/box.py", line 514, in __getattr__
    value = self.__getitem__(item, _ignore_default=True)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/box/box.py", line 509, in __getitem__
    raise BoxKeyError(str(err)) from _exception_cause(err)
box.exceptions.BoxKeyError: "'logging'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/box/box.py", line 516, in __getattr__
    value = object.__getattribute__(self, item)
AttributeError: 'Config' object has no attribute 'logging'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/test_slack.py", line 1, in <module>
    from prefect import Flow, task
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/__init__.py", line 1, in <module>
    import prefect.utilities
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/__init__.py", line 1, in <module>
    import prefect.utilities.logging
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/logging.py", line 280, in <module>
    context.logger = prefect_logger = configure_logging()
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/logging.py", line 277, in configure_logging
    return _create_logger(name)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/logging.py", line 249, in _create_logger
    context.config.logging.format, context.config.logging.datefmt
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/box/box.py", line 530, in __getattr__
    raise BoxKeyError(str(err)) from _exception_cause(err)
box.exceptions.BoxKeyError: "'Config' object has no attribute 'logging'"
k

Kevin Kho

03/16/2022, 1:34 AM
Let me try logging out and seeing what happens one sec
I see why it’s trying to authenticate. The 
slack_notifier
is coded to fetch the secret from Prefect Cloud. This is an old piece of code and the built-in SlackTask is preferred. You should be able to run this locally without Prefect Cloud:
Copy code
from prefect import Flow, task
from prefect.tasks.notifications.slack_task import SlackTask

def mystatehandler(obj, old_state, new_state):
    SlackTask().run(message="test message", webhook_secret="SLACK_WEBHOOK_URL")
    return new_state

@task(name="1/x task", state_handlers=[mystatehandler])
def div(x):
    return 1 / x


@task(name="Add 1 task", state_handlers=[mystatehandler])
def add(x):
    return x + 1


with Flow('Add Divide Flow') as f:
    res = div(x=add(-1))


final_state = f.run()
But it still doesn’t tell me why your config has no logging. That’s really weird. For that, something seems wrong with Prefect. I would suggest reinstalling
k

Kevin Otte

03/16/2022, 1:41 AM
Ok, thanks for clarifying - is there a doc or community post referencing this other code?
also, feels like we're getting closer but I get this now:
Copy code
(venv39) ➜  airflow-pipelines git:(minor_refactor) ✗ python test_slack.py
[2022-03-15 21:40:08-0400] INFO - prefect.FlowRunner | Beginning Flow run for 'Add Divide Flow'
[2022-03-15 21:40:08-0400] INFO - prefect.TaskRunner | Task 'Add 1 task': Starting task run...
[2022-03-15 21:40:08-0400] ERROR - prefect.TaskRunner | Unexpected error while calling state handlers: HTTPError('403 Client Error: Forbidden for url: <https://hooks.slack.com/services/T015STTHK0A/B0342AESY6L/>')
Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/kevin/starkiller/research/airflow-pipelines/test_slack.py", line 5, in mystatehandler
    SlackTask().run(message="test message", webhook_secret="SLACK_WEBHOOK_URL")
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/tasks/notifications/slack_task.py", line 66, in run
    r.raise_for_status()
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: <https://hooks.slack.com/services/T015STTHK0A/B0342AESY6L/>
[2022-03-15 21:40:08-0400] INFO - prefect.TaskRunner | Task 'Add 1 task': Finished task run for task with final state: 'Failed'
[2022-03-15 21:40:08-0400] INFO - prefect.TaskRunner | Task '1/x task': Starting task run...
[2022-03-15 21:40:08-0400] ERROR - prefect.TaskRunner | Unexpected error while calling state handlers: HTTPError('403 Client Error: Forbidden for url: <https://hooks.slack.com/services/T015STTHK0A/B0342AESY6L/>')
Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/kevin/starkiller/research/airflow-pipelines/test_slack.py", line 5, in mystatehandler
    SlackTask().run(message="test message", webhook_secret="SLACK_WEBHOOK_URL")
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/tasks/notifications/slack_task.py", line 66, in run
    r.raise_for_status()
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: <https://hooks.slack.com/services/T015STTHK0A/B0342AESY6L/>
[2022-03-15 21:40:08-0400] INFO - prefect.TaskRunner | Task '1/x task': Finished task run for task with final state: 'Failed'
[2022-03-15 21:40:08-0400] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
(venv39) ➜  airflow-pipelines git:(minor_refactor) ✗
k

Kevin Kho

03/16/2022, 1:43 AM
That looks a lot better on the Prefect side. Seems you don’t have permissions to post to the webhooks. State Handler documentation is here. Why don’t you modify the message, use my webhook here:
Copy code
SLACK_WEBHOOK_URL = "<https://hooks.slack.com/services/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>"
and then i’ll tell you if I get the message. At that point we know the code is working fine haha
SlackTask docs is here. It is preferred over 
slack_notifier
because you can customize the message
k

Kevin Otte

03/16/2022, 1:46 AM
haha I ran the code but my guess is it didnt work
k

Kevin Kho

03/16/2022, 1:46 AM
Oof yeah I got no message. What was your error? Still forbidden?
k

Kevin Otte

03/16/2022, 1:46 AM
Copy code
(venv39) ➜  airflow-pipelines git:(minor_refactor) ✗ python test_slack.py
[2022-03-15 21:45:43-0400] INFO - prefect.FlowRunner | Beginning Flow run for 'Add Divide Flow'
[2022-03-15 21:45:43-0400] INFO - prefect.TaskRunner | Task 'Add 1 task': Starting task run...
[2022-03-15 21:45:43-0400] ERROR - prefect.TaskRunner | Unexpected error while calling state handlers: ValueError('Local Secret "<https://hooks.slack.com/services/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>" was not found.')
Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/kevin/starkiller/research/airflow-pipelines/test_slack.py", line 5, in mystatehandler
    SlackTask().run(message="test message", webhook_secret= "<https://hooks.slack.com/services/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>")
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/tasks/notifications/slack_task.py", line 61, in run
    webhook_url = webhook_url or cast(str, Secret(webhook_secret).get())
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/secrets.py", line 167, in get
    raise ValueError(
ValueError: Local Secret "<https://hooks.slack.com/services/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>" was not found.
[2022-03-15 21:45:43-0400] INFO - prefect.TaskRunner | Task 'Add 1 task': Finished task run for task with final state: 'Failed'
[2022-03-15 21:45:43-0400] INFO - prefect.TaskRunner | Task '1/x task': Starting task run...
[2022-03-15 21:45:43-0400] ERROR - prefect.TaskRunner | Unexpected error while calling state handlers: ValueError('Local Secret "<https://hooks.slack.com/services/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>" was not found.')
Traceback (most recent call last):
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/runner.py", line 161, in handle_state_change
    new_state = self.call_runner_target_handlers(old_state, new_state)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 113, in call_runner_target_handlers
    new_state = handler(self.task, old_state, new_state) or new_state
  File "/Users/kevin/starkiller/research/airflow-pipelines/test_slack.py", line 5, in mystatehandler
    SlackTask().run(message="test message", webhook_secret= "<https://hooks.slack.com/services/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>")
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/utilities/tasks.py", line 456, in method
    return run_method(self, *args, **kwargs)
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/tasks/notifications/slack_task.py", line 61, in run
    webhook_url = webhook_url or cast(str, Secret(webhook_secret).get())
  File "/Users/kevin/starkiller/research/airflow-pipelines/venv39/lib/python3.9/site-packages/prefect/client/secrets.py", line 167, in get
    raise ValueError(
ValueError: Local Secret "<https://hooks.slack.com/services/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>" was not found.
[2022-03-15 21:45:43-0400] INFO - prefect.TaskRunner | Task '1/x task': Finished task run for task with final state: 'Failed'
[2022-03-15 21:45:43-0400] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
(venv39) ➜  airflow-pipelines git:(minor_refactor) ✗
k

Kevin Kho

03/16/2022, 1:48 AM
Ah so 
SlackTask
takes in the 
webhook_secret
which is a secret name. So you should be changing your secret in the config.toml and in the code still have`webhook_secret="SLACK_WEBHOOK_URL"`. You shouldn’t pass the webhook in there.
I got them. “test message”
k

Kevin Otte

03/16/2022, 1:50 AM
haha yeah
ok
great I just got it to work on my end
haha
thanks!!
k

Kevin Kho

03/16/2022, 1:50 AM
So Prefect Server has no Vault but Prefect Cloud has one so if you store your secrets there, it will be pulled during runtime for task execution. If you are on server, then you just need to set the local secrets
Nice! Of course!
5 Views
Powered by