Igor Bondartsov
04/07/2021, 3:55 PMKevin Kho
HOST
variable a Prefect Parameter
. Is that right?Igor Bondartsov
04/07/2021, 4:31 PMKevin Kho
Kevin Kho
Flow
runs. Only the task.run
method will be able to take Parameters
. If you look at the code behind the task, some of the attributes can be overwritten during runtime. Unfortunately for this Task, the HOST
can not be overwrriten. This is a simple fix of editing the Task definition to allow HOST
to be overwrriten and if would be a welcome Pull Request (if you’re interested). Otherwise, the current workaround is something like the following:
from prefect import Flow, Task, Parameter, task
class TestTask(Task):
def __init__(self, p: str, **kwargs):
self.p = p
super().__init__(**kwargs)
def run(self):
self.logger.warning(f"Printing the parameter {self.p}")
@task
def override_task(override_p):
task = TestTask(override_p)
task.run()
return
with Flow('test') as flow:
p = Parameter('p', default="aaa")
override_task(p)
flow.run(parameters=dict(p="testing"))
Kevin Kho
Task
that instantiates and calls the run method of the MySQL
task. This override_task can now take the Parameter
.Domantas
04/08/2021, 2:26 PMfrom prefect import Flow, Task, Parameter, task
class TestTask(Task):
def __init__(self, p: str, **kwargs):
self.p = p
super().__init__(**kwargs)
def run(self):
mysql_response = MySQLExecute(
name="testing_connection",
db_name="",
user="",
host=self.p,
password='',
port=1234
)
self.logger.warning(f"Printing the parameter {self.p}")
return mysql_response
@task
def override_task(override_p):
task = TestTask(override_p)
return task.run()
with Flow('test') as flow:
p = Parameter('p', default="aaa")
override_task(p)
flow.run(parameters=dict(p="host_name"))
Even declaring empty, incorrect values for db password, user, db_name, etc, I got my tasks run with no error:
[2021-04-08 17:21:32+0300] INFO - prefect.TaskRunner | Task 'override_task': Finished task run for task with final state: 'Success'
Could you elaborate more on how to implement MySQL execution, because I'm clearly misunderstanding this case.Kevin Kho
MySQLExecute.run()
because this code is just initializing it. The run method is the one that will execute itKevin Kho
override_task
, just put the MySQLExecute task in there instead of TestTask
and then call the run()
and that should workDomantas
04/08/2021, 2:45 PM