Hey again, so I am obviously doing something wrong...
# ask-community
a
Hey again, so I am obviously doing something wrong since I seem to be unable to create a flow that takes in host/port/etc for a PostgresExecute task at runtime rather than definition time since those are not available during definition time for me...
k
Hi @Adam Brusselback! You are not doing anything wrong and this is a known issue. We want to make tasks params passable at runtime. This is the issue for that.
Just a friendly reminder to post the code blocks in the thread is possible to make the main channel a bit neater.
a
Will do in the future, sorry about that.
k
Maybe you can ping the guy who said he’d work on this task, or you can copy the Postgres task you need and edit it to suit your needs. It’s actually easy once you see the source for the task
No problem at all!
a
Ah, so those init ones are simply not available if you need to init at runtime? So it's only possible to set those init values for any task that doesn't allow reconfiguring those params at runtime when you are originally building the flow, then they are locked in?
There has to be a little more structure to what is possible to do when/where that is documented somewhere? I seem to have a hard time figuring that out.
k
Task libraries are mostly community contributed and that’s why there is some inconsistency but we’re working to standardize them. Some other tasks in the task library let you override the
init
params by passing them to the
run
params. For this specific task, yes that is the case that you can’t reconfigure. Yes they do seem locked in I think you’re right. Our goal is to make everything available at runtime with the task library.
#You can modify the task from the task library like this: class PostgresExecute(Task): “”" Task for executing a query against a Postgres database. Args: - **kwargs (dict, optional): additional keyword arguments to pass to the Task constructor “”" def __init__( self, **kwargs ): super().__init__(**kwargs) def run( self, db_name: str = None, user: str = None, host: str = None, port: int = None, query: str = None, data: tuple = None, commit: bool = False, password: str = None, ): “”" Task run method. Executes a query against Postgres database. Args: - query (str, optional): query to execute against database - data (tuple, optional): values to use in query, must be specified using placeholder is query string - commit (bool, optional): set to True to commit transaction, defaults to false - password (str): password used to authenticate; should be provided from a
Secret
task Returns: - None Raises: - ValueError: if query parameter is None or a blank string - DatabaseError: if exception occurs when executing the query “”" if not query: raise ValueError(“A query string must be provided”) # connect to database, open cursor # allow psycopg2 to pass through any exceptions raised conn = pg.connect( dbname=db_name, user=user, password=password, host=host, port=port, ) # try to execute query # context manager automatically rolls back failed transactions try: with conn, conn.cursor() as cursor: executed = cursor.execute(query=query, vars=data) if commit: conn.commit() else: conn.rollback() return executed # ensure connection is closed finally: conn.close()
I think something like this will work for your use case
a
Makes sense. Appreciate the help here. That whole "what is available / usable" when the flow is "built" vs "run" seems like a part that could use some more words in the the documentation, but you very much helped me understand. Appreciate the modified task, will give it a shot / tweak as necessary!
k
Ideally, we just get all the tasks to be run-time modifiable but yes I understand the lack of clarity.