https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Adam

09/17/2020, 9:32 AM
Hello everyone! Hope you’re having a great day so far! I need some quick help regarding the API for
PostgresFetch
. It seems to require
user
and
host
arguments when declaring the task and then
password
can be supplied when instantiating it. How can I send all when I instantiate it rather? I’m trying to achieve something like this:
Copy code
get_customers = PostgresFetch(
    name="Get customers ready for migration",
    db_name="postgres",
    user="CAN_I_USE_ENV_SECRET_HERE?",
    host="CAN_I_USE_ENV_SECRET_HERE?",
    query="""
        SELECT DISTINCT customer_id
        FROM customer_migration_request
    """,
    fetch="all",
)


with Flow("migrate customers") as flow:
    postgres_user = EnvVarSecret("POSTGRES_USER", raise_if_missing=True)
    postgres_password = EnvVarSecret("POSTGRES_PASSWORD", raise_if_missing=True)
    postgres_host = EnvVarSecret("POSTGRES_HOST", raise_if_missing=True)

    customers = get_customers(
        password=postgres_password, user=postgres_user, host=postgres_host
    )
The above throws an error that user and host are invalid arguments to
get_customers
d

Dylan

09/17/2020, 1:52 PM
Hi @Adam! It looks like you’ll either need to: 1. Instantiate
get_customers
inside your flow context so your secrets are defined:
Copy code
with Flow("migrate customers") as flow:
    postgres_user = EnvVarSecret("POSTGRES_USER", raise_if_missing=True)
    postgres_password = EnvVarSecret("POSTGRES_PASSWORD", raise_if_missing=True)
    postgres_host = EnvVarSecret("POSTGRES_HOST", raise_if_missing=True)

    customers = PostgresFetch(
    name="Get customers ready for migration",
    db_name="postgres",
    user=postgres_user,
    host=postgres_host,
    password=postgres_password,
    query="""
        SELECT DISTINCT customer_id
        FROM customer_migration_request
    """,
    fetch="all",
    )()
Or you’ll need to define your secrets outside of your flow context
Copy code
postgres_user = EnvVarSecret("POSTGRES_USER", raise_if_missing=True)
postgres_password = EnvVarSecret("POSTGRES_PASSWORD", raise_if_missing=True)
postgres_host = EnvVarSecret("POSTGRES_HOST", raise_if_missing=True)

get_customers = PostgresFetch(
    name="Get customers ready for migration",
    db_name="postgres",
    user=postgres_user,
    host=postgres_host,
    password=postgres_password,
    query="""
        SELECT DISTINCT customer_id
        FROM customer_migration_request
    """,
    fetch="all",
)



with Flow("migrate customers") as flow:
    customers = get_customers()
a

Adam

09/17/2020, 4:10 PM
Thanks @Dylan, makes sense. Which is the more idiomatic way?
d

Dylan

09/17/2020, 4:26 PM
I’ve seen both! I don’t think an idiom has developed yet
a

Adam

09/18/2020, 11:46 AM
So it seems neither options above are working for me @Dylan. In both cases it seems that the
PostgresFetch
task is not handling the
EnvVarSecret
task properly. It seems to stringify it:
Copy code
[2020-09-18 11:45:38] ERROR - prefect.TaskRunner | Unexpected error: OperationalError('could not translate host name "<Task: POSTGRES_HOST>" to address: nodename nor servname provided, or not known\n')
Any ideas @Dylan and team?