hey all im trying to pass a task s result to a function that Prefect Community #prefect-community

hey all, im trying to pass a task's result to a fu...

Jon

10/27/2022, 6:49 PM

hey all, im trying to pass a task's result to a function that's not a task. I'm getting:

TypeError: not all arguments converted during string formatting

is there a way to get the parameter's value without giving function with a task decorator?

✅ 1

Nate

10/27/2022, 6:54 PM

Hi @Jon Since the

prefect.Parameter

is a special type of

task

object, it must be used in a flow context - so this approach may not work what are you trying to accomplish by passing parameters to regular python functions?

Jon

10/27/2022, 6:55 PM

thanks @Nate. i just made an edit -- it's a task's result, not a parameter

Jon

10/27/2022, 6:55 PM

we are web scraping, and have some utility function, like for logging in.

Copy code

@task
def get_login_credentials():
       ...

@task
def extract_with_headless_browser(login_credentials):
        ... some logic to spin up a browser ...

        # not a task, just some imported fn
        page_logged_in = browser_utils.login(
            page,
            LOGIN_URL,
            USERNAME_LABEL,
            PASSWORD_LABEL,
            SIGN_IN_BUTTON_TEXT,
            login_credentials["username"],
            login_credentials["password"],
        )

        ... scrape something ...

with Flow() as f:
    credentials = get_login_credentials()
    extract_with_headless_browser(credentials)

Jon

10/27/2022, 6:58 PM

ideally

browser_utils.login

would have a task decorator, but doesn't seem we can serialize a headless browser 😅

Nate

10/27/2022, 7:02 PM

so it looks like

credentials

is some dict-like return value of

get_login_credentials

, which becomes

login_credentials

in the scope of

extract_with_headless_browser

, which looks totally fine to me can you show the stack trace? I'm wondering which line actually threw that error

Jon

10/27/2022, 7:03 PM

Copy code

Task 'extract_electricity_amr_reports': Exception encountered during task execution!
Traceback (most recent call last):
  File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "<string>", line 39, in extract_electricity_amr_reports
  File "/Users/jonyoung/Dev/workflows/workflows/common/browser_utils.py", line 81, in login
    <http://logger.info|logger.info>(username, password)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1446, in info
    self._log(INFO, msg, args, **kwargs)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/Users/jonyoung/Dev/workflows/.venv/lib/python3.9/site-packages/prefect/utilities/logging.py", line 184, in emit
    msg = self.format(record)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 663, in format
    record.message = record.getMessage()
  File "/Users/jonyoung/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 367, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting

Jon

10/27/2022, 7:04 PM

that stack trace shows the logger unhappy about

username

and

password

Jon

10/27/2022, 7:04 PM

i think same error w/o using the logger ..

Nate

10/27/2022, 7:07 PM

so I reproduced that error just by passing multiple args to our logger like

Copy code

from prefect import Flow
import prefect

with Flow('f') as f:
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello", "World")

f.run()

Copy code

File "/Users/nate/opt/miniconda3/envs/prefect-1/lib/python3.8/logging/__init__.py", line 373, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting

can you just do

<http://logger.info|logger.info>(f"{username=} {password=}")

instead of passing multiple args?

Jon

10/27/2022, 7:09 PM

that fixed it

Jon

10/27/2022, 7:09 PM

thanks 🤦

Nate

10/27/2022, 7:10 PM

great to hear 😄 sure thing

Jon

10/27/2022, 7:12 PM

appreciate it

5 Views

Open in Slack

Previous Next