https://prefect.io logo
r

Robert Bastian

12/15/2020, 3:08 PM
Hi Prefect Community. I have a question related to the "right" way to do things in Prefect. Here is an example task using Postgres and Secrets. It works but doesn't seem "correct" to me since both PostgresFetch and Secrets are subclasses of Task. Any advice would be appreciated!
Copy code
def get_record_count():
    logger = prefect.context.get("logger")
    pg = PostgresFetch(host='<http://hostname.us-east-1.rds.amazonaws.com|hostname.us-east-1.rds.amazonaws.com>', db_name='testdatabase', user='test')
    pg.query="select count(*) from testdatabase.test_table"
    secret = PrefectSecret('TEST_PASSWORD')
    password = secret.run()
    result = pg.run(password=password)
    return result[0]
j

Jim Crist-Harif

12/15/2020, 3:19 PM
Hi Robert, You can wrap tasks in tasks by manually calling the
run
method (as you've done above). An alternate way would be to have two tasks - one for getting the secret, and one for running the query. This lets you compose things from the task library better without having to call
run
yourself. Something like the following (untested):
Copy code
with Flow("your-flow") as flow:
    password = PrefectSecret("TEST_PASSWORD")
    pg = PostgresFetch(
        host='<http://hostname.us-east-1.rds.amazonaws.com|hostname.us-east-1.rds.amazonaws.com>',
        db_name='testdatabase',
        user='test',
        query="select count(*) from testdatabase.test_table"
    )
    result = pg(password=password)
👀 1
r

Robert Bastian

12/15/2020, 6:51 PM
@Jim Crist-Harif Thanks Jim, that works great. My assumption is that Prefect is executing the run method on the tasks "behind the scenes" when it sees the dependencies as part of the Flow context? Thx again!
j

Jim Crist-Harif

12/15/2020, 6:59 PM
If a task is added to the flow, its
run
method is a called at flow runtime. In your first example, you wrote a function that manually created to
Task
objects (but didn't add them to a flow), then called their
run
methods manually. In this case, they're more like helper functions you're using to do your work. The former (compose tasks into flows) is the recommended way to use prefect, but the latter (use parts of our task library to build your own tasks) also works.
Also, slack etiquette note - there's no reason to also send your response to the channel, your question will still be picked up in-thread. No harm, just fyi.