andres aava
01/14/2022, 6:40 AMclass saveChartmogulActivities(Task):
def __init__(self, apikey, **kwargs):
self.apikey = apikey
self.endpoint = ENDPOINT_CHARTMOGUL_ACTIVITIES_EXPORT
self.tmp_file_path = TMP_FILE_PATH
super().__init__(**kwargs)
def run(self):
return self.apikey
apikey seems to be passed as Task not as actual value.
Flow looks like this, it seems to be something with my custom defined Task class, but could not figure it out or find quickly from community here.
Could anybody tell just by looking at it? :)
with Flow('chartmogul-extract', run_config=RUN_CONFIG) as flow:
# Reads Secrets from Prefect Cloud > Team > Secrets
chartmogul_secrets = PrefectSecret("CHARTMOGUL")
chartmogul_apikey = chartmogul_secrets['api_key']
# Extract
cm_activities_task = saveChartmogulActivities(chartmogul_apikey, log_stdout=True)
cm_activities_run = cm_activities_task()
Although when I pass apikey as hardcoded string value, it works.emre
01/14/2022, 9:07 AMSecret
tasks retrieve secrets during flow run, not while you are initializing your tasks and building your flows. You should retrieve any secrets you want with Secret
tasks, and pass them to other task via their run
method, not __init__
emre
01/14/2022, 9:10 AMclass saveChartmogulActivities(Task):
def __init__(self, **kwargs):
self.endpoint = ENDPOINT_CHARTMOGUL_ACTIVITIES_EXPORT
self.tmp_file_path = TMP_FILE_PATH
super().__init__(**kwargs)
def run(self, apikey):
return apikey
and this:
with Flow('chartmogul-extract', run_config=RUN_CONFIG) as flow:
# Reads Secrets from Prefect Cloud > Team > Secrets
chartmogul_secrets = PrefectSecret("CHARTMOGUL")
chartmogul_apikey = chartmogul_secrets['api_key']
# Extract
cm_activities_run = saveChartmogulActivities(log_stdout=True)(apikey=chartmogul_apikey)
Note the double parentheses with your custom task. The first one initializes your Task, basically __init__
. The second one binds your task to the flow, and other tasks that exist in the flow, so that during flow.run()
your task gets passed the correct upstream task outputs.andres aava
01/14/2022, 9:38 AMcm_activities_task = saveChartmogulActivities(log_stdout=True)(apikey=chartmogul_apikey)
cm_activities_run = cm_activities_task()
Or could it also be like this below?
cm_activities_task = saveChartmogulActivities(log_stdout=True)
cm_activities_run = cm_activities_task(apikey=chartmogul_apikey)
emre
01/14/2022, 10:13 AM