https://prefect.io logo
Channels
Powered by
p

Peyton Runyan

02/16/2021, 10:58 PM
Is there a way to define a function at runtime? I have a sql task that I would like to pass a username and password at runtime, and I'm struggling pretty hard to get it moving. More in the thread
Copy code
with Flow("config_flow") as f:

    mode = Parameter("Mode", default="dev")

    user, db, password_var  = set_creds(mode)
    print_task(user)
    print_task(db)
    print_task(password_var)
    password_task = EnvVarSecret()
    password = password_task(password_var)

    # Get our database items
    # thing = Parameter("thing", default=["Thing 1"])
    # d = dog(thing)

    sql_task = SqlServerFetch(
            db_name=db,
            user=user,
            host=prefect.config.sql_server.server,
            query=get_manual_override_rows,
            fetch='many',
            fetch_count=3,
            result=result_formatter,
            name="Database Query"
            # commit: bool = False,
    )

    s = sql_task(password=password)
From the logs:
It's reading in the config variables well enough for the print task
But fails for the SQL task (it passes a task instead of the output of the task)
a

Amanda Wee

02/16/2021, 11:20 PM
I may be mistaken, but shouldn't this:
Copy code
password_task = EnvVarSecret()
password = password_task(password_var)
be:
Copy code
password = EnvVarSecret(password_var)
?
🙏 1
c

Chris White

02/16/2021, 11:24 PM
Hi @Peyton Runyan, Amanda might be right to resolve this specific error but more broadly this sort of confusion is common with people new to deferred computation; I highly recommend organizing your code as follows:
Copy code
## all task initializations / etc. go here
mode = Parameter("Mode", default="dev")
sql_task = SQLServer(...) # etc.

## all task _dependencies_ are set here
with Flow("config_flow") as f:
    ...
This should help you reason through issues like this more cleanly
🙏 1
p

Peyton Runyan

02/16/2021, 11:31 PM
Thanks both of yall!
👍 1
Is there a good "idiot's guide" to thinking about these things that I can grind through to get me up to speed? I swear I've been through all of the docs, but it seems like I'm short a framework to scaffold things onto effectively. I'm a coder with fairly limited devops experience. Or is this more of a "try, fail, ask, repeat" situation?
c

Chris White

02/16/2021, 11:54 PM
In theory this document has all the information you may need for the build vs. runtime distinction: https://docs.prefect.io/core/advanced_tutorials/task-guide.html
✅ 1
Powered by